Video Communication Including Holographic Content

ABSTRACT

A video communication system uses a light field display to present a holographic image of a remote scene (e.g., a hologram of a remote participant). The system may include a local light field display assembly and a controller. The controller generates display instructions based on visual data corresponding to a remote scene received from a remote image capture system (e.g., a remote light field display system). The display instructions cause the local light field display assembly to generate a holographic image of the remote scene.

This application is related to International Application Nos.PCT/US2017/042275, PCT/US2017/042276, PCT/US2017/042418,PCT/US2017/042452, PCT/US2017/042462, PCT/US2017/042466,PCT/US2017/042467, PCT/US2017/042468, PCT/US2017/042469,PCT/US2017/042470, and PCT/US2017/042679, all of which are incorporatedby reference herein in their entirety.

BACKGROUND

The present disclosure relates generally to communication systems, andin particular to video communication using light field display systems.

As the availability of communications bandwidth has gone up and the costof digital cameras has gone down, video has become an increasinglypopular method of communication. However, several factors limit thequality of user experience with existing video communication solutions.Typically, the camera and screen are located close together but atnoticeably different locations. As a result, participants do not makeeye contact and often do not pick up on other gestures and expressionsthat humans use to add context to the words spoken. Direct interactionis also difficult as the displays on which video is presented act as abarrier between participants. While some existing video communicationtechnology allows documents to be viewed through a screen-sharingfunction, this results in the video of participants being removed orreduced in size.

SUMMARY

A video communication system uses a light field display to present aholographic image of a remote scene, which may include one or moreremote participants. In one embodiment, the system includes a locallight field display assembly and a controller. The controller generatesdisplay instructions based on visual data corresponding to a remotescene received from a remote image capture system, such as another lightfield display assembly or a remote light field display system. Thedisplay instructions cause the local light field display assembly togenerate a holographic image of the remote scene.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram of a light field display module presenting aholographic object, in accordance with one or more embodiments.

FIG. 2A is a cross section of a portion of a light field display module,in accordance with one or more embodiments.

FIG. 2B is a cross section of a portion of a light field display module,in accordance with one or more embodiments.

FIG. 3A is a perspective view of a light field display module, inaccordance with one or more embodiments.

FIG. 3B is a cross-sectional view of a light field display module whichincludes interleaved energy relay devices, in accordance with one ormore embodiments.

FIG. 4A is a perspective view of portion of a light field display systemthat is tiled in two dimensions to form a single-sided seamless surfaceenvironment, in accordance with one or more embodiments.

FIG. 4B is a perspective view of a portion of light field display systemin a multi-sided seamless surface environment, in accordance with one ormore embodiments.

FIG. 4C is a top-down view of a light field display system with anaggregate surface in a winged configuration, in accordance with one ormore embodiments.

FIG. 4D is a side view of a light field display system with an aggregatesurface in a sloped configuration, in accordance with one or moreembodiments.

FIG. 4E is a top-down view of a light field display system with anaggregate surface on a front wall of a room, in accordance with one ormore embodiments.

FIG. 4F is a side view of a side view of a light field display systemwith an aggregate surface on the front wall of the room, in accordancewith one or more embodiments.

FIG. 5 is a block diagram of a light field display system, in accordancewith one or more embodiments.

FIG. 6 is an illustration of an example light field display system forvideo conferencing, in accordance with one or more embodiments.

FIG. 7 is an illustration of an alternative configuration of a videoconferencing space, in accordance with one or more embodiments.

FIG. 8A is an illustration of a light field display system presentingholographic content including a holographic video chat participant, inaccordance with one or more embodiments.

FIG. 8B is an illustration of the light field display system of FIG. 8Apresenting holographic content including an image of the video chatparticipant, in accordance with one or more embodiments.

FIG. 9 is an illustration of a LF display system presenting holographiccontent including representations of participants in a group video chat,in accordance with one or more embodiments.

DETAILED DESCRIPTION

A light field (LF) display system provides video communication such asvideo conferencing, video chat, or pre-recorded video messages. Thevideo communication includes holographic content representing a remotescene, such as holographic images of remote participants, holographicimages of props, and holographic whiteboards. Although the terms “videoconferencing” and “video chat” are used for convenience to refer to moreformal and less formal communication, respectively, any functionalitydescribed with reference to one may be provided in the other.

In various embodiments, a holographic image of a participant in onelocation is presented to a participant in a different location. Theholographic image provides a three-dimensional (3D) representation ofthe participant that can be viewed without the need for glasses, aheadset, or other viewing equipment. Where the communication is live, abidirectional surface that both emits and absorbs light may be used.Thus, there may be a one-to-one correspondence between gaze directionsfor participants in both locations, enabling the participants to makeeye contact as if they were in the same physical space. Furthermore, thepresence of a 3D image may aid participants in picking up on gesturesand/or expressions that communicate information that might be missedwith conventional video communication. Consequently, the use of the LFdisplay system may provide participants with the impression that theyare located in the same space, even when separated by thousands ofmiles.

In some embodiments, additional holographic images may be provided toimprove and/or facilitate the communication experience. For example,holographic props such as product prototypes may be provided to furtherfacilitate communication as if the participants were located in the samespace. Similarly, a holographic whiteboard may be provided on whichparticipants may draw, and the contents may be synchronized between twoor more locations. Because the whiteboard is holographic, it need not beconstrained to a two-dimensional (2D) surface. In one embodiment,participants may draw in 3D within an area of space (e.g., a box)designated as a virtual whiteboard. As another example, participants maybe able to alter their own appearance and/or the appearance of otherparticipants. This may include partial changes, such as changingclothes, hair color, lighting, or the like as well as complete changes,such as representing the participant with an avatar whose motion ismapped to that of the corresponding participant.

Holographic content presented by the LF display system may also beaugmented with other sensory stimuli (e.g., tactile and/or audio). Forexample, ultrasonic sources in the LF display system may projectultrasonic pressure waves that create a volumetric haptic projection.The volumetric haptic projection provides a tactile surface thatcorresponds to some or all of the holographic objects that areprojected. Holographic content may also include additional visualcontent (i.e., 2D or 3D visual content). The coordination of energysources that enables a cohesive experience is part of the LF system inimplementations with multiple energy sources (i.e., holographic objectsproviding the correct haptic feel and sensory stimuli at any given pointin time). For example, a LF system may include a controller tocoordinate presentation of holographic content and haptic surfaces.

In some embodiments, the LF display system may include elements thatenable the system to project at least one type of energy, and,simultaneously, sense at least one type of energy. Sensed energy may beused for recording how a viewer responds to the holographic content. Forexample, a LF display system can project both holographic objects forviewing as well as ultrasonic waves for haptic perception, andsimultaneously record imaging information for tracking of viewers (e.g.,video conference participants) and other scene analysis. As an example,such a system may project a holographic product prototype that aparticipant may manipulate via touch (e.g., by reaching out and rotatingthe holographic prototype to view it from a different angle), with thisinteraction with the holographic prototype recorded by the LF displaysystem. The LF display system components that perform energy sensing ofthe environment may be integrated into the display surface, or they maybe dedicated sensors that are separate from the display surface.

Light Field Display System Overview

FIG. 1 is a diagram 100 of a light field (LF) display module 110presenting a holographic object 120, in accordance with one or moreembodiments. The LF display module 110 is part of a light field (LF)display system. The LF display system presents holographic contentincluding at least one holographic object using one or more LF displaymodules. The LF display system can present holographic content to one ormultiple viewers. In some embodiments, the LF display system may alsoaugment the holographic content with other sensory content (e.g., touch,audio, smell, temperature, etc.). For example, as discussed below, theprojection of focused ultrasonic sound waves may generate a mid-airtactile sensation that can simulate a surface of some or all of aholographic object. The LF display system includes one or more LFdisplay modules 110, and is discussed in detail below with regard toFIGS. 2-9.

The LF display module 110 is a holographic display that presentsholographic objects (e.g., the holographic object 120) to one or moreviewers (e.g., viewer 140). The LF display module 110 includes an energydevice layer (e.g., an emissive electronic display or acousticprojection device) and an energy waveguide layer (e.g., optical lensarray). Additionally, the LF display module 110 may include an energyrelay layer for the purpose of combining multiple energy sources ordetectors together to form a single surface. At a high-level, the energydevice layer generates energy (e.g., holographic content) that is thendirected using the energy waveguide layer to a region in space inaccordance with one or more four-dimensional (4D) light field functions.The LF display module 110 may also project and/or sense one or moretypes of energy simultaneously. For example, LF display module 110 maybe able to project a holographic image as well as an ultrasonic tactilesurface in a viewing volume, while simultaneously detecting imaging datafrom the viewing volume. The operation of the LF display module 110 isdiscussed in more detail below with regard to FIGS. 2-3.

The LF display module 110 generates holographic objects within aholographic object volume 160 using one or more 4D light field functions(e.g., derived from a plenoptic function). The holographic objects canbe three-dimensional (3D), two-dimensional (2D), or some combinationthereof. Moreover, the holographic objects may be polychromatic (e.g.,full color). The holographic objects may be projected in front of thescreen plane, behind the screen plane, or split by the screen plane. Aholographic object 120 can be presented such that it is perceivedanywhere within the holographic object volume 160. A holographic objectwithin the holographic object volume 160 may appear to a viewer 140 tobe floating in space.

A holographic object volume 160 represents a volume in which holographicobjects may be perceived by a viewer 140. The holographic object volume160 can extend in front of the surface of the display area 150 (i.e.,towards the viewer 140) such that holographic objects can be presentedin front of the plane of the display area 150. Additionally, theholographic object volume 160 can extend behind the surface of thedisplay area 150 (i.e., away from the viewer 140), allowing forholographic objects to be presented as if they are behind the plane ofthe display area 150. In other words, the holographic object volume 160may include all the rays of light that originate (e.g., are projected)from a display area 150 and can converge to create a holographic object.Herein, light rays may converge at a point that is in front of thedisplay surface, at the display surface, or behind the display surface.More simply, the holographic object volume 160 encompasses all of thevolume from which a holographic object may be perceived by a viewer.

A viewing volume 130 is a volume of space from which holographic objects(e.g., holographic object 120) presented within a holographic objectvolume 160 by the LF display system are fully viewable. The holographicobjects may be presented within the holographic object volume 160, andviewed within a viewing volume 130, such that they are indistinguishablefrom actual objects. A holographic object is formed by projecting thesame light rays that would be generated from the surface of the objectwere it physically present.

In some cases, the holographic object volume 160 and the correspondingviewing volume 130 may be relatively small—such that it is designed fora single viewer. In other embodiments, as discussed in detail below withregard to, e.g., FIGS. 4 and 6-9, the LF display modules may be enlargedand/or tiled to create larger holographic object volumes andcorresponding viewing volumes that can accommodate a large range ofviewers (e.g., one to thousands). The LF display modules presented inthis disclosure may be built so that the full surface of the LF displaycontains holographic imaging optics, with no inactive or dead space, andwithout any need for bezels. In these embodiments, the LF displaymodules may be tiled so that the imaging area is continuous across theseam between LF display modules, and the bond line between the tiledmodules is virtually undetectable using the visual acuity of the eye.Notably, in some configurations, some portion of the display surface maynot include holographic imaging optics, although they are not describedin detail herein.

The flexible size and/or shape of a viewing volume 130 allows forviewers to be unconstrained within the viewing volume 130. For example,a viewer 140 can move to a different position within a viewing volume130 and see a different view of the holographic object 120 from thecorresponding perspective. To illustrate, referring to FIG. 1, theviewer 140 is at a first position relative to the holographic object 120such that the holographic object 120 appears to be a head-on view of adolphin. The viewer 140 may move to other locations relative to theholographic object 120 to see different views of the dolphin. Forexample, the viewer 140 may move such that he/she sees a left side ofthe dolphin, a right side of the dolphin, etc., much like if the viewer140 was looking at an actual dolphin and changed his/her relativeposition to the actual dolphin to see different views of the dolphin. Insome embodiments, the holographic object 120 is visible to all viewerswithin the viewing volume 130 that have an unobstructed line (i.e., notblocked by an object/person) of sight to the holographic object 120.These viewers may be unconstrained such that they can move around withinthe viewing volume to see different perspectives of the holographicobject 120. Accordingly, the LF display system may present holographicobjects such that a plurality of unconstrained viewers maysimultaneously see different perspectives of the holographic objects inreal-world space as if the holographic objects were physically present.

In contrast, conventional displays (e.g., stereoscopic, virtual reality,augmented reality, or mixed reality) generally require each viewer towear some sort of external device (e.g., 3-D glasses, a near-eyedisplay, or a head-mounted display) in order to see content.Additionally and/or alternatively, conventional displays may requirethat a viewer be constrained to a particular viewing position (e.g., ina chair that has fixed location relative to the display). For example,when viewing an object shown by a stereoscopic display, a viewer alwaysfocuses on the display surface, rather than on the object, and thedisplay will always present just two views of an object that will followa viewer who attempts to move around that perceived object, causingdistortions in the perception of that object. With a light fielddisplay, however, viewers of a holographic object presented by the LFdisplay system do not need to wear an external device, nor be confinedto a particular position, in order to see the holographic object. The LFdisplay system presents the holographic object in a manner that isvisible to viewers in much the same way a physical object would bevisible to the viewers, with no requirement of special eyewear, glasses,or a head-mounted accessory. Further, the viewer may view holographiccontent from any location within a viewing volume.

Notably, potential locations for holographic objects within theholographic object volume 160 are limited by the size of the volume. Inorder to increase the size of the holographic object volume 160, a sizeof a display area 150 of the LF display module 110 may be increasedand/or multiple LF display modules may be tiled together in a mannerthat forms a seamless display surface. The seamless display surface hasan effective display area that is larger than the display areas of theindividual LF display modules. Some embodiments relating to tiling LFdisplay modules are discussed below with regard to FIGS. 4, and 6-9. Asillustrated in FIG. 1, the display area 150 is rectangular resulting ina holographic object volume 160 that is a pyramid. In other embodiments,the display area may have some other shape (e.g., hexagonal), which alsoaffects the shape of the corresponding viewing volume.

Additionally, while the above discussion focuses on presenting theholographic object 120 within a portion of the holographic object volume160 that is between the LF display module 110 and the viewer 140, the LFdisplay module 110 can additionally present content in the holographicobject volume 160 behind the plane of the display area 150. For example,the LF display module 110 may make the display area 150 appear to be asurface of the ocean that the holographic object 120 is jumping out of.And the displayed content may be such that the viewer 140 is able tolook through the displayed surface to see marine life that is under thewater. Moreover, the LF display system can generate content thatseamlessly moves around the holographic object volume 160, includingbehind and in front of the plane of the display area 150.

FIG. 2A illustrates across section 200 of a portion of a LF displaymodule 210, in accordance with one or more embodiments. The LF displaymodule 210 may be the LF display module 110. In other embodiments, theLF display module 210 may be another LF display module with a differentdisplay area shape than display area 150. In the illustrated embodiment,the LF display module 210 includes an energy device layer 220, an energyrelay layer 230, and an energy waveguide layer 240. Some embodiments ofthe LF display module 210 have different components than those describedhere. For example, in some embodiments, the LF display module 210 doesnot include the energy relay layer 230. Similarly, the functions can bedistributed among the components in a different manner than is describedhere.

The display system described here presents an emission of energy thatreplicates the energy normally surrounding an object in the real world.Here, emitted energy is directed towards a specific direction from everycoordinate on the display surface. In other words, the variouscoordinates on the display surface act as projection locations foremitted energy. The directed energy from the display surface enablesconvergence of many rays of energy, which, thereby, can createholographic objects. For visible light, for example, the LF display willproject a very large number of light rays from the projection locationsthat may converge at any point in the holographic object volume so theywill appear to come from the surface of a real-world object located inthis region of space from the perspective of a viewer that is locatedfurther away than the object being projected. In this way, the LFdisplay is generating the rays of reflected light that would leave suchan object's surface from the perspective of the viewer. The viewerperspective may change on any given holographic object, and the viewerwill see a different view of that holographic object.

The energy device layer 220 includes one or more electronic displays(e.g., an emissive display such as an OLED) and one or more other energyprojection and/or energy receiving devices as described herein. The oneor more electronic displays are configured to display content inaccordance with display instructions (e.g., from a controller of a LFdisplay system). The one or more electronic displays include a pluralityof pixels, each with an intensity that is individually controlled. Manytypes of commercial displays, such as emissive LED and OLED displays,may be used in the LF display.

The energy device layer 220 may also include one or more acousticprojection devices and/or one or more acoustic receiving devices. Anacoustic projection device generates one or more pressure waves thatcomplement the holographic object 250. The generated pressure waves maybe, e.g., audible, ultrasonic, or some combination thereof. An array ofultrasonic pressure waves may be used for volumetric tactile sensation(e.g., at a surface of the holographic object 250). An audible pressurewave is used for providing audio content (e.g., immersive audio) thatcan complement the holographic object 250. For example, assuming theholographic object 250 is a dolphin, one or more acoustic projectiondevices may be used to (1) generate a tactile surface that is collocatedwith a surface of the dolphin such that viewers may touch theholographic object 250; and (2) provide audio content corresponding tonoises a dolphin makes such as clicks, chirping, or chatter. An acousticreceiving device (e.g., a microphone or microphone array) may beconfigured to monitor ultrasonic and/or audible pressure waves within alocal area of the LF display module 210.

The energy device layer 220 may also include one or more imagingsensors. An imaging sensor may be sensitive to light in a visibleoptical band, and in some cases may be sensitive to light in other bands(e.g., infrared). The imaging sensor may be, e.g., a complementary metaloxide semi-conductor (CMOS) array, a charged coupled device (CCD), anarray of photodetectors, some other sensor that captures light, or somecombination thereof. The LF display system may use data captured by theone or more imaging sensor for position location tracking of viewers.

In some configurations, the energy relay layer 230 relays energy (e.g.,electromagnetic energy, mechanical pressure waves, etc.) between theenergy device layer 220 and the energy waveguide layer 240. The energyrelay layer 230 includes one or more energy relay elements 260. Eachenergy relay element includes a first surface 265 and a second surface270, and it relays energy between the two surfaces. The first surface265 of each energy relay element may be coupled to one or more energydevices (e.g., electronic display or acoustic projection device). Anenergy relay element may be composed of, e.g., glass, carbon, opticalfiber, optical film, plastic, polymer, or some combination thereof.Additionally, in some embodiments, an energy relay element may adjustmagnification (increase or decrease) of energy passing between the firstsurface 265 and the second surface 270. If the relay offersmagnification, then the relay may take the form of an array of bondedtapered relays, called tapers, where the area of one end of the tapermay be substantially larger than the opposite end. The large end of thetapers can be bonded together to form a seamless energy surface 275. Oneadvantage is that space is created on the multiple small ends of eachtaper to accommodate the mechanical envelope of multiple energy sources,such as the bezels of multiple displays. This extra room allows theenergy sources to be placed side-by-side on the small taper side, witheach energy source having their active areas directing energy into thesmall taper surface and relayed to the large seamless energy surface.Another advantage to using tapered relays is that there is nonon-imaging dead space on the combined seamless energy surface formed bythe large end of the tapers. No border or bezel exists, and so theseamless energy surfaces can then be tiled together to form a largersurface with virtually no seams according to the visual acuity of theeye.

The second surfaces of adjacent energy relay elements come together toform an energy surface 275. In some embodiments, a separation betweenedges of adjacent energy relay elements is less than a minimumperceptible contour as defined by a visual acuity of a human eye having,for example, 20/40 vision, such that the energy surface 275 iseffectively seamless from the perspective of a viewer 280 within aviewing volume 285.

In some embodiments, the second surfaces of adjacent energy relayelements are fused together with processing steps that may include oneor more of pressure, heat, and a chemical reaction, in such a way noseam exists between them. And still in other embodiments, an array ofenergy relay elements is formed by molding one side of a continuousblock of relay material into an array of small taper ends, eachconfigured to transport energy from an energy device attached to thesmall tapered end into a single combined surface with a larger areawhich is never subdivided.

In some embodiments, one or more of the energy relay elements exhibitenergy localization, where the energy transport efficiency in thelongitudinal direction substantially normal to the surfaces 265 and 270is much higher than the transport efficiency in the perpendiculartransverse plane, and where the energy density is highly localized inthis transverse plane as the energy wave propagates between surface 265and surface 270. This localization of energy allows an energydistribution, such as an image, to be efficiency relayed between thesesurfaces without any significant loss in resolution.

The energy waveguide layer 240 directs energy from a location (e.g., acoordinate) on the energy surface 275 into a specific energy propagationpath outward from the display surface into the holographic viewingvolume 285 using waveguide elements in the energy waveguide layer 240.The energy propagation path is defined by two angular dimensionsdetermined at least by the energy surface coordinate location relativeto the waveguide. The waveguide is associated with a spatial 2Dcoordinate. Together, these four coordinates form a four-dimensional(4D) energy field. As an example, for electromagnetic energy, thewaveguide elements in the energy waveguide layer 240 direct light frompositions on the seamless energy surface 275 along different propagationdirections through the viewing volume 285. In various examples, thelight is directed in accordance with a 4D light field function to formthe holographic object 250 within the holographic object volume 255.

Each waveguide element in the energy waveguide layer 240 may be, forexample, a lenslet composed of one or more elements. In someconfigurations, the lenslet may be a positive lens. The positive lensmay have a surface profile that is spherical, aspherical, or freeform.Additionally, in some embodiments, some or all of the waveguide elementsmay include one or more additional optical components. An additionaloptical component may be, e.g., an energy-inhibiting structure such as abaffle, a positive lens, a negative lens, a spherical lens, anaspherical lens, a freeform lens, a liquid crystal lens, a liquid lens,a refractive element, a diffractive element, or some combinationthereof. In some embodiments, the lenslet and/or at least one of theadditional optical components is able to dynamically adjust its opticalpower. For example, the lenslet may be a liquid crystal lens or a liquidlens. Dynamic adjustment of a surface profile the lenslet and/or atleast one additional optical component may provide additionaldirectional control of light projected from a waveguide element.

In the illustrated example, the holographic object volume 255 of the LFdisplay has boundaries formed by light ray 256 and light ray 257, butcould be formed by other rays. The holographic object volume 255 is acontinuous volume that extends both in front (i.e., towards the viewer280) of the energy waveguide layer 240 and behind it (i.e., away fromthe viewer 280). In the illustrated example, ray 256 and ray 257 areprojected from opposite edges of the LF display module 210 at thehighest angle relative to the normal to the display surface 277 that maybe perceived by a user, but these could be other projected rays. Therays define the field-of-view of the display, and, thus, define theboundaries for the holographic viewing volume 285. In some cases, therays define a holographic viewing volume where the full display can beobserved without vignetting (e.g., an ideal viewing volume). As thefield of view of the display increases, the convergence point of ray 256and ray 257 will be closer to the display. Thus, a display having alarger field of view allows a viewer 280 to see the full display at acloser viewing distance. Additionally, ray 256 and 257 may form an idealholographic object volume. Holographic objects presented in an idealholographic object volume can be seen anywhere in the viewing volume285.

In some examples, holographic objects may be presented to only a portionof the viewing volume 285. In other words, holographic object volumesmay be divided into any number of viewing sub-volumes (e.g., viewingsub-volume 290). Additionally, holographic objects can be projectedoutside of the holographic object volume 255. For example, holographicobject 251 is presented outside of holographic object volume 255.Because the holographic object 251 is presented outside of theholographic object volume 255 it cannot be viewed from every location inthe viewing volume 285. For example, holographic object 251 may bevisible from a location in viewing sub-volume 290, but not visible fromthe location of the viewer 280.

For example, we turn to FIG. 2B to illustrate viewing holographiccontent from different viewing sub-volumes. FIG. 2B illustrates a crosssection 200 of a portion of a LF display module, in accordance with oneor more embodiments. The cross-section of FIG. 2B is the same as thecross-section of FIG. 2A. However, FIG. 2B illustrates a different setof light rays projected from the LF display module 210. Ray 256 and ray257 still form a holographic object volume 255 and a viewing volume 285.However, as shown, rays projected from the top of the LF display module210 and the bottom of the LF display module 210 overlap to form variousviewing sub-volumes (e.g., view sub-volumes 290A, 290B, 290C, and 290D)within the viewing volume 285. A viewer in the first viewing sub-volume(e.g., 290A) may be able to perceive holographic content presented inthe holographic object volume 255 that viewers in the other viewingsub-volumes (e.g., 290B, 290C, and 290D) are unable to perceive.

More simply, as illustrated in FIG. 2A, holographic object volume 255 isa volume in which holographic objects may be presented by LF displaysystem such that they may be perceived by viewers (e.g., viewer 280) inviewing volume 285. In this way, the viewing volume 285 is an example ofan ideal viewing volume, while the holographic object volume 255 is anexample of an ideal object volume. However, in various configurations,viewers may perceive holographic objects presented by LF display system200 in other example holographic object volumes. More generally, an“eye-line guideline” applies when viewing holographic content projectedfrom an LF display module. The eye-line guideline asserts that the lineformed by a viewer's eye position and a holographic object being viewedmust intersect a LF display surface.

When viewing holographic content presented by the LF display module 210,each eye of the viewer 280 sees a different perspective of theholographic object 250 because the holographic content is presentedaccording to a 4D light field function. Moreover, as the viewer 280moves within the viewing volume 285 he/she would also see differentperspectives of the holographic object 250 as would other viewers withinthe viewing volume 285. As will be appreciated by one of ordinary skillin the art, a 4D light field function is well known in the art and willnot be elaborated further herein.

As described in more detail herein, in some embodiments, the LF displaycan project more than one type of energy. For example, the LF displaymay project two types of energy, such as, for example, mechanical energyand electromagnetic energy. In this configuration, energy relay layer230 may include two separate energy relays which are interleavedtogether at the energy surface 275, but are separated such that theenergy is relayed to two different energy device layers 220. Here, onerelay may be configured to transport electromagnetic energy, whileanother relay may be configured to transport mechanical energy. In someembodiments, the mechanical energy may be projected from locationsbetween the electromagnetic waveguide elements on the energy waveguidelayer 240, helping form structures that inhibit light from beingtransported from one electromagnetic waveguide element to another. Insome embodiments, the energy waveguide layer 240 may also includewaveguide elements that transport focused ultrasound along specificpropagation paths in accordance with display instructions from acontroller.

Note that in alternate embodiments (not shown), the LF display module210 does not include the energy relay layer 230. In this case, theenergy surface 275 is an emission surface formed using one or moreadjacent electronic displays within the energy device layer 220. And insome embodiments, with no energy relay layer, a separation between edgesof adjacent electronic displays is less than a minimum perceptiblecontour as defined by a visual acuity of a human eye having 20/40vision, such that the energy surface is effectively seamless from theperspective of the viewer 280 within the viewing volume 285.

LF Display Modules

FIG. 3A is a perspective view of a LF display module 300A, in accordancewith one or more embodiments. The LF display module 300A may be the LFdisplay module 110 and/or the LF display module 210. In otherembodiments, the LF display module 300A may be some other LF displaymodule. In the illustrated embodiment, the LF display module 300Aincludes an energy device layer 310, and energy relay layer 320, and anenergy waveguide layer 330. The LF display module 300A is configured topresent holographic content from a display surface 365 as describedherein. For convenience, the display surface 365 is illustrated as adashed outline on the frame 390 of the LF display module 300A, but is,more accurately, the surface directly in front of waveguide elementsbounded by the inner rim of the frame 390. The display surface 365includes a plurality of projection locations from which energy can beprojected. Some embodiments of the LF display module 300A have differentcomponents than those described here. For example, in some embodiments,the LF display module 300A does not include the energy relay layer 320.Similarly, the functions can be distributed among the components in adifferent manner than is described here.

The energy device layer 310 is an embodiment of the energy device layer220. The energy device layer 310 includes four energy devices 340 (threeare visible in the figure). The energy devices 340 may all be the sametype (e.g., all electronic displays), or may include one or moredifferent types (e.g., includes electronic displays and at least oneacoustic energy device).

The energy relay layer 320 is an embodiment of the energy relay layer230. The energy relay layer 320 includes four energy relay devices 350(three are visible in the figure). The energy relay devices 350 may allrelay the same type of energy (e.g., light), or may relay one or moredifferent types (e.g., light and sound). Each of the relay devices 350includes a first surface and a second surface, the second surface of theenergy relay devices 350 being arranged to form a singular seamlessenergy surface 360. In the illustrated embodiment, each of the energyrelay devices 350 is tapered such that the first surface has a smallersurface area than the second surface, which allows accommodation for themechanical envelopes of the energy devices 340 on the small end of thetapers. This also allows the seamless energy surface to be borderless,since the entire area can project energy. This means that this seamlessenergy surface can be tiled by placing multiple instances of LF displaymodule 300A together, without dead space or bezels, so that the entirecombined surface is seamless. In other embodiments, the first surfaceand the second surface have the same surface area.

The energy waveguide layer 330 is an embodiment of the energy waveguidelayer 240. The energy waveguide layer 330 includes a plurality ofwaveguide elements 370. As discussed above with respect to FIG. 2, theenergy waveguide layer 330 is configured to direct energy from theseamless energy surface 360 along specific propagation paths inaccordance with a 4D light field function to form a holographic object.Note that in the illustrated embodiment the energy waveguide layer 330is bounded by a frame 390. In other embodiments, there is no frame 390and/or a thickness of the frame 390 is reduced. Removal or reduction ofthickness of the frame 390 can facilitate tiling the LF display module300A with additional LF display modules.

Note that in the illustrated embodiment, the seamless energy surface 360and the energy waveguide layer 330 are planar. In alternate embodiments,not shown, the seamless energy surface 360 and the energy waveguidelayer 330 may be curved in one or more dimensions.

The LF display module 300A can be configured with additional energysources that reside on the surface of the seamless energy surface, andallow the projection of an energy field in additional to the lightfield. In one embodiment, an acoustic energy field may be projected fromelectrostatic speakers (not illustrated) mounted at any number oflocations on the seamless energy surface 360. Further, the electrostaticspeakers of the LF display module 300A are positioned within the lightfield display module 300A such that the dual-energy surfacesimultaneously projects sound fields and holographic content. Forexample, the electrostatic speakers may be formed with one or morediaphragm elements that are transmissive to some wavelengths ofelectromagnetic energy, and driven with one or more conductive elements(e.g., planes which sandwich the one or more diaphragm elements). Theelectrostatic speakers may be mounted on to the seamless energy surface360, so that the diaphragm elements cover some of the waveguideelements. The conductive electrodes of the speakers may be co-locatedwith structures designed to inhibit light transmission betweenelectromagnetic waveguides, and/or located at positions betweenelectromagnetic waveguide elements (e.g., frame 390). In variousconfigurations, the speakers can project an audible sound and/or manysources of focused ultrasonic energy that produces a haptic surface.

In some configurations an energy device 340 may sense energy. Forexample, an energy device may be a microphone, a light sensor, anacoustic transducer, etc. As such, the energy relay devices may alsorelay energy from the seamless energy surface 360 to the energy devicelayer 310. That is, the seamless energy surface 360 of the LF displaymodule forms a bidirectional energy surface when the energy devices andenergy relay devices 340 are configured to simultaneously emit and senseenergy (e.g., emit light fields and sense sound).

More broadly, an energy device 340 of a LF display module 340 can beeither an energy source or an energy sensor. The LF display module 300Acan include various types of energy devices that act as energy sourcesand/or energy sensors to facilitate the projection of high qualityholographic content to a user. Other sources and/or sensors may includethermal sensors or sources, infrared sensors or sources, image sensorsor sources, mechanical energy transducers that generate acoustic energy,feedback sources, etc. Many other sensors or sources are possible.Further, the LF display modules can be tiled such that the LF displaymodule can form an assembly that projects and senses multiple types ofenergy from a large aggregate seamless energy surface

In various embodiments of LF display module 300A, the seamless energysurface 360 can have various surface portions where each surface portionis configured to project and/or emit specific types of energy. Forexample, when the seamless energy surface is a dual-energy surface, theseamless energy surface 360 includes one or more surface portions thatproject electromagnetic energy, and one or more other surface portionsthat project ultrasonic energy. The surface portions that projectultrasonic energy may be located on the seamless energy surface 360between electromagnetic waveguide elements, and/or co-located withstructures designed to inhibit light transmission betweenelectromagnetic waveguide elements. In an example where the seamlessenergy surface is a bidirectional energy surface, the energy relay layer320 may include two types of energy relay devices interleaved at theseamless energy surface 360. In various embodiments, the seamless energysurface 360 may be configured such that portions of the surface underany particular waveguide element 370 are all energy sources, all energysensors, or a mix of energy sources and energy sensors.

FIG. 3B is a cross-sectional view of a LF display module 300B whichincludes interleaved energy relay devices, in accordance with one ormore embodiments. Energy relay device 350A transports energy between theenergy relay first surface 345A connected to energy device 340A, and theseamless energy surface 360. Energy relay 350B transports energy betweenthe energy relay first surface 345B connected to energy device 340B, andthe seamless energy surface 360. Both relay devices are interleaved atinterleaved energy relay device 352, which is connected to the seamlessenergy surface 360. In this configuration, surface 360 containsinterleaved energy locations of both energy devices 340A and 340B, whichmay be energy sources or energy sensors. Accordingly, the LF displaymodule 300B may be configured as either a dual energy projection devicefor projecting more than one type of energy, or as a bidirectionalenergy device for simultaneously projecting one type of energy andsensing another type of energy. The LF display module 300B may be the LFdisplay module 110 and/or the LF display module 210. In otherembodiments, the LF display module 300B may be some other LF displaymodule.

The LF display module 300B includes many components similarly configuredto those of LF display module 300A in FIG. 3A. For example, in theillustrated embodiment, the LF display module 300B includes an energydevice layer 310, energy relay layer 320, a seamless energy surface 360,and an energy waveguide layer 330 including at least the samefunctionality of those described in regard to FIG. 3A. Additionally, theLF display module 300B may present and/or receive energy from thedisplay surface 365. Notably, the components of the LF display module300B are alternatively connected and/or oriented than those of the LFdisplay module 300A in FIG. 3A. Some embodiments of the LF displaymodule 300B have different components than those described here.Similarly, the functions can be distributed among the components in adifferent manner than is described here. FIG. 3B illustrates the designof a single LF display module 300B that may be tiled to produce a dualenergy projection surface or a bidirectional energy surface with alarger area.

In an embodiment, the LF display module 300B is a LF display module of abidirectional LF display system. A bidirectional LF display system maysimultaneously project energy and sense energy from the display surface365. The seamless energy surface 360 contains both energy projecting andenergy sensing locations that are closely interleaved on the seamlessenergy surface 360. Therefore, in the example of FIG. 3B, the energyrelay layer 320 is configured in a different manner than the energyrelay layer of FIG. 3A. For convenience, the energy relay layer of LFdisplay module 300B will be referred to herein as the “interleavedenergy relay layer.”

The interleaved energy relay layer 320 includes two legs: a first energyrelay device 350A and a second energy relay device 350B. Each of thelegs are illustrated as a lightly shaded area in FIG. 3B. Each of thelegs may be made of a flexible relay material, and formed with asufficient length to use with energy devices of various sizes andshapes. In some regions of the interleaved energy relay layer, the twolegs are tightly interleaved together as they approach the seamlessenergy surface 360. In the illustrated example, the interleaved energyrelay devices 352 are illustrated as a darkly shaded area.

While interleaved at the seamless energy surface 360, the energy relaydevices are configured to relay energy to/from different energy devices.The energy devices are at energy device layer 310. As illustrated,energy device 340A is connected to energy relay device 350A and energydevice 340B is connected to energy relay device 350B. In variousembodiments, each energy device may be an energy source or energysensor.

An energy waveguide layer 330 includes waveguide elements 370 to steerenergy waves from the seamless energy surface 360 along projected pathstowards a series of convergence points. In this example, a holographicobject 380 is formed at the series of convergence points. Notably, asillustrated, the convergence of energy at the holographic object 380occurs on the viewer side (i.e., the front side) of the display surface365. However, in other examples, the convergence of energy may beanywhere in the holographic object volume, which extends both in frontof the display surface 365 and behind the display surface 365. Thewaveguide elements 370 can simultaneously steer incoming energy to anenergy device (e.g., an energy sensor), as described below.

In one example embodiment of LF display module 300B, an emissive displayis used as an energy source (e.g., energy device 340A) and an imagingsensor is used as an energy sensor (e.g., energy device 340B). In thismanner, the LF display module 300B can simultaneously projectholographic content and detect light from the volume in front of thedisplay surface 365. In this manner, this embodiment of the LF displaymodule 300B functions as both a LF display and an LF sensor.

In an embodiment, the LF display module 300B is configured tosimultaneously project a light field from projection locations on thedisplay surface to the front of the display surface and capture a lightfield from the front of the display surface at the projection locations.In this embodiment, the energy relay device 350A connects a first set oflocations at the seamless energy surface 360 positioned under thewaveguide elements 370 to an energy device 340A. In an example, energydevice 340A is an emissive display having an array of source pixels. Theenergy relay device 3408 connects a second set of locations at theseamless energy surface 360 positioned under waveguide elements 370 toan energy device 340B. In an example, the energy device 340B is animaging sensor having an array of sensor pixels. The LF display module300B may be configured such that the locations at the seamless energysurface 365 that are under a particular waveguide element 370 are allemissive display locations, all imaging sensor locations, or somecombination of these locations. In other embodiments, the bidirectionalenergy surface can project and receive various other forms of energy.

In another example embodiment of the LF display module 300B, the LFdisplay module is configured to project two different types of energy.For example, in an embodiment, energy device 340A is an emissive displayconfigured to emit electromagnetic energy and energy device 340B is anultrasonic transducer configured to emit mechanical energy. As such,both light and sound can be projected from various locations at theseamless energy surface 360. In this configuration, energy relay device350A connects the energy device 340A to the seamless energy surface 360and relays the electromagnetic energy. The energy relay device isconfigured to have properties (e.g. varying refractive index) which makeit efficient for transporting electromagnetic energy. Energy relaydevice 350B connects the energy device 340B to the seamless energysurface 360 and relays mechanical energy. Energy relay device 350B isconfigured to have properties for efficient transport of ultrasoundenergy (e.g. distribution of materials with different acousticimpedance). In some embodiments, the mechanical energy may be projectedfrom locations between the waveguide elements 370 on the energywaveguide layer 330. The locations that project mechanical energy mayform structures that serve to inhibit light from being transported fromone electromagnetic waveguide element to another. In one example, aspatially separated array of locations that project ultrasonicmechanical energy can be configured to create three-dimensional hapticshapes and surfaces in mid-air. The surfaces may coincide with projectedholographic objects (e.g., holographic object 380). In some examples,phase delays and amplitude variations across the array can assist increating the haptic shapes.

In various embodiments, the LF display module 300B with interleavedenergy relay devices may include multiple energy device layers with eachenergy device layer including a specific type of energy device. In theseexamples, the energy relay layers are configured to relay theappropriate type of energy between the seamless energy surface 360 andthe energy device layer 310.

Tiled LF Display Modules

FIG. 4A is a perspective view of a portion of LF display system 400 thatis tiled in two dimensions to form a single-sided seamless surfaceenvironment, in accordance with one or more embodiments. The LF displaysystem 400 includes a plurality of LF display modules that are tiled toform an array 410. More explicitly, each of the small squares in thearray 410 represents a tiled LF display module 412. The LF displaymodule 412 may be the same as LF display module 300A or 300B. The array410 may cover, for example, some or all of a surface (e.g., a wall) of aroom. The LF array may cover other surfaces, such as, for example, atable top, a cubicle divider, etc.

The array 410 may project one or more holographic objects. For example,in the illustrated embodiment, the array 410 projects a holographicobject 420 and a holographic object 422. Tiling of the LF displaymodules 412 allows for a much larger viewing volume as well as allowsfor objects to be projected out farther distances from the array 410.For example, in the illustrated embodiment, the viewing volume is,approximately, the entire area in front of and behind the array 410rather than a localized volume in front of (and behind) a LF displaymodule 412.

In some embodiments, the LF display system 400 presents the holographicobject 420 to a viewer 430 and a viewer 434. The viewer 430 and theviewer 434 receive different perspectives of the holographic object 420.For example, the viewer 430 is presented with a direct view of theholographic object 420, whereas the viewer 434 is presented with a moreoblique view of the holographic object 420. As the viewer 430 and/or theviewer 434 move, they are presented with different perspectives of theholographic object 420. This allows a viewer to visually interact with aholographic object by moving relative to the holographic object. Forexample, as the viewer 430 walks around a holographic object 420, theviewer 430 sees different sides of the holographic object 420 as long asthe holographic object 420 remains in the holographic object volume ofthe array 410. Accordingly, the viewer 430 and the viewer 434 maysimultaneously see the holographic object 420 in real-world space as ifit is truly there. Additionally, the viewer 430 and the viewer 434 donot need to wear an external device in order to see the holographicobject 420, as the holographic object 420 is visible to viewers in muchthe same way a physical object would be visible. Additionally, here, theholographic object 422 is illustrated behind the array because theviewing volume of the array extends behind the surface of the array. Inthis manner, the holographic object 422 may be presented to the viewer430 and/or viewer 434.

In some embodiments, the LF display system 400 may include a trackingsystem that tracks positions of the viewer 430 and the viewer 434. Insome embodiments, the tracked position is the position of a viewer. Inother embodiments, the tracked position is that of the eyes of a viewer.The position tracking of the eye is different from gaze tracking whichtracks where an eye is looking (e.g., uses orientation to determine gazelocation). The eyes of the viewer 430 and the eyes of the viewer 434 arein different locations.

In various configurations, the LF display system 400 may include one ormore tracking systems. For example, in the illustrated embodiment ofFIG. 4A, LF display system includes a tracking system 440 that isexternal to the array 410. Here, the tracking system may be a camerasystem coupled to the array 410. External tracking systems are describedin more detail in regard to FIG. 5. In other example embodiments, thetracking system may be incorporated into the array 410 as describedherein. For example, an energy device (e.g., energy device 340) of oneor more LF display modules 412 containing a bidirectional energy surfaceincluded in the array 410 may be configured to capture images of viewersin front of the array 410. In whichever case, the tracking system(s) ofthe LF display system 400 determines tracking information about theviewers (e.g., viewer 430 and/or viewer 434) viewing holographic contentpresented by the array 410.

Tracking information describes a position in space (e.g., relative tothe tracking system) for the position of a viewer, or a position of aportion of a viewer (e.g. one or both eyes of a viewer, or theextremities of a viewer). A tracking system may use any number of depthdetermination techniques to determine tracking information. The depthdetermination techniques may include, e.g., structured light, time offlight, stereo imaging, some other depth determination technique, orsome combination thereof. The tracking system may include varioussystems configured to determine tracking information. For example, thetracking system may include one or more infrared sources (e.g.,structured light sources), one or more imaging sensors that can captureimages in the infrared (e.g., red-blue-green-infrared camera), and aprocessor executing tracking algorithms. The tracking system may use thedepth estimation techniques to determine positions of viewers. In someembodiments, the LF display system 400 generates holographic objectsbased on tracked positions, motions, or gestures of the viewer 430and/or the viewer 434 as described herein. For example, the LF displaysystem 400 may generate a holographic object responsive to a viewercoming within a threshold distance of the array 410 and/or a particularposition.

The LF display system 400 may present one or more holographic objectsthat are customized to each viewer based in part on the trackinginformation. For example, the viewer 430 may be presented with theholographic object 420, but not the holographic object 422. Similarly,the viewer 434 may be presented with the holographic object 422, but notthe holographic object 420. For example, the LF display system 400tracks a position of each of the viewer 430 and the viewer 434. The LFdisplay system 400 determines a perspective of a holographic object thatshould be visible to a viewer based on their position relative to wherethe holographic object is to be presented. The LF display system 400selectively projects light from specific pixels that correspond to thedetermined perspective. Accordingly, the viewer 434 and the viewer 430can simultaneously have experiences that are, potentially, completelydifferent. In other words, the LF display system 400 may presentholographic content to viewing sub-volumes of the viewing volume (i.e.,similar to the viewing sub-volumes 290A, 290B, 290C, and 290D shown inFIG. 2B). For example, as illustrated, because the LF display system 400can track the position of the viewer 430, the LF display system 400 maypresent space content (e.g., holographic object 420) to a viewingsub-volume surrounding the viewer 430 and safari content (e.g.,holographic object 422) to a viewing sub-volume surrounding the viewer434. In contrast, conventional systems would have to use individualheadsets to provide a similar experience.

In some embodiments the LF display system 400 may include one or moresensory feedback systems. The sensory feedback systems provide othersensory stimuli (e.g., tactile, audio, or smell) that augment theholographic objects 420 and 422. For example, in the illustratedembodiment of FIG. 4A, the LF display system 400 includes a sensoryfeedback system 442 external to the array 410. In one example, thesensory feedback system 442 may be an electrostatic speaker coupled tothe array 410. External sensory feedback systems are described in moredetail in regard to FIG. 5. In other example embodiments, the sensoryfeedback system may be incorporated into the array 410 as describedherein. For example, an energy device (e.g., energy device 340A in FIG.3B) of a LF display module 412 included in the array 410 may beconfigured to project ultrasonic energy to viewers in front of the arrayand/or receive imaging information from viewers in front of the array.In whichever case, the sensory feedback system presents and/or receivessensory content to/from the viewers (e.g., viewer 430 and/or viewer 434)viewing holographic content (e.g., holographic object 420 and/orholographic objected 422) presented by the array 410.

The LF display system 400 may include a sensory feedback system 442 thatincludes one or more acoustic projection devices external to the array.Alternatively or additionally, the LF display system 400 may include oneor more acoustic projection devices integrated into the array 410 asdescribed herein. The acoustic projection devices may consist of anarray of ultrasonic sources configured to project a volumetric tactilesurface. In some embodiments, the tactile surface may be coincident witha holographic object (e.g., at a surface of the holographic object 420)for one or more surfaces of a holographic object if a portion of aviewer gets within a threshold distance of the one or more surfaces. Thevolumetric tactile sensation may allow the user to touch and feelsurfaces of the holographic object. The plurality of acoustic projectiondevices may also project an audible pressure wave that provides audiocontent (e.g., immersive audio) to viewers. Accordingly, the ultrasonicpressure waves and/or the audible pressure waves can act to complement aholographic object.

In various embodiments, the LF display system 400 may provide othersensory stimuli based in part on a tracked position of a viewer. Forexample, the holographic object 422 illustrated in FIG. 4A is a lion,and the LF display system 400 may have the holographic object 422 roarboth visually (i.e., the holographic object 422 appears to roar) andaudibly (i.e., one or more acoustic projection devices project apressure wave that the viewer 430 perceives as a lion's roar emanatingfrom the holographic object 422.

Note that, in the illustrated configuration, the holographic viewingvolume may be limited in a manner similar to the viewing volume 285 ofthe LF display system 200 in FIG. 2. This can limit the amount ofperceived immersion that a viewer will experience with a single walldisplay unit. One way to address this is to use multiple LF displaymodules that are tiled along multiple sides as described below withrespect to FIG. 4B-4F.

FIG. 4B is a perspective view of a portion of a LF display system 402 ina multi-sided seamless surface environment, in accordance with one ormore embodiments. The LF display system 402 is substantially similar tothe LF display system 400 except that the plurality of LF displaymodules are tiled to create a multi-sided seamless surface environment.More specifically, the LF display modules are tiled to form an arraythat is a six-sided aggregated seamless surface environment. In FIG. 4B,the plurality of LF display modules cover all the walls, the ceiling,and the floor of a room. In other embodiments, the plurality of LFdisplay modules may cover some, but not all of a wall, a floor, aceiling, or some combination thereof. In other embodiments, a pluralityof LF display modules are tiled to form some other aggregated seamlesssurface. For example, the walls may be curved such that a cylindricalaggregated energy environment is formed. Moreover, as described belowwith regard to FIGS. 6-9, in some embodiments, the LF display modulesmay be tiled to form a surface in a conference room or office (e.g.,walls, etc.).

The LF display system 402 may project one or more holographic objects.For example, in the illustrated embodiment the LF display system 402projects the holographic object 420 into an area enclosed by thesix-sided aggregated seamless surface environment. In this example, theviewing volume of the LF display system is also contained within thesix-sided aggregated seamless surface environment. Note that, in theillustrated configuration, the viewer 434 may be positioned between theholographic object 420 and a LF display module 414 that is projectingenergy (e.g., light and/or pressure waves) that is used to form theholographic object 420. Accordingly, the positioning of the viewer 434may prevent the viewer 430 from perceiving the holographic object 420formed from energy from the LF display module 414. However, in theillustrated configuration there is at least one other LF display module,e.g., a LF display module 416, that is unobstructed (e.g., by the viewer434) and can project energy to form the holographic object 420 and beobserved by viewer 430. In this manner, occlusion by viewers in thespace can cause some portion of the holographic projections todisappear, but the effect is much less than if only one side of thevolume was populated with holographic display panels. Holographic object422 is illustrated “outside” the walls of the six-sided aggregatedseamless surface environment because the holographic object volumeextends behind the aggregated surface. Thus, the viewer 430 and/or theviewer 434 can perceive the holographic object 422 as “outside” of theenclosed six-sided environment which they can move throughout.

As described above in reference to FIG. 4A, in some embodiments, the LFdisplay system 402 actively tracks positions of viewers and maydynamically instruct different LF display modules to present holographiccontent based on the tracked positions. Accordingly, a multi-sidedconfiguration can provide a more robust environment (e.g., relative toFIG. 4A) for providing holographic objects where unconstrained viewersare free to move throughout the area enclosed by the multi-sidedseamless surface environment.

Notably, various LF display systems may have different configurations.Further, each configuration may have a particular orientation ofsurfaces that, in aggregate, form a seamless display surface (“aggregatesurface”). That is, the LF display modules of a LF display system can betiled to form a variety of aggregate surfaces. For example, in FIG. 4B,the LF display system 402 includes LF display modules tiled to form asix-sided aggregate surface that approximates the walls of a room. Insome other examples, an aggregate surface may only occur on a portion ofa surface (e.g., half of a wall) rather than a whole surface (e.g., anentire wall). Some examples are described herein.

In some configurations, the aggregate surface of a LF display system mayinclude an aggregate surface configured to project energy towards alocalized viewing volume. Projecting energy to a localized viewingvolume allows for a higher quality viewing experience by, for example,increasing the density of projected energy in a specific viewing volume,increasing the FOV for the viewers in that volume, and bringing theviewing volume closer to the display surface.

For example, FIG. 4C illustrates top down view of a LF display system450A with an aggregate surface in a “winged” configuration. In thisexample, the LF display system 450A is located in a room with a frontwall 452, a rear wall 454, a first sidewall 456, a second sidewall 458,a ceiling (not shown), and a floor (not shown). The first sidewall 456,the second sidewall 458, the rear wall 454, floor, and the ceiling areall orthogonal. The LF display system 450A includes LF display modulestiled to form an aggregate surface 460 covering the front wall. Thefront wall 452, and thus the aggregate surface 460, includes threeportions: (i) a first portion 462 approximately parallel with the rearwall 454 (i.e., a central surface), (ii) a second portion 464 connectingthe first portion 462 to the first sidewall 456 and placed at an angleto project energy towards the center of the room (i.e., a first sidesurface), and (iii) a third portion 466 connecting the first portion 462to the second sidewall 458 and placed at an angle to project energytowards the center of the room (i.e., a second side surface). The firstportion is a vertical plane in the room and has a horizontal and avertical axis. The second and third portions are angled towards thecenter of the room along the horizontal axis.

In this example, the viewing volume 468A of the LF display system 450Ais in the center of the room and partially surrounded by the threeportions of the aggregate surface 460. An aggregate surface that atleast partially surrounds a viewer (“surrounding surface”) increases theimmersive experience of the viewers.

To illustrate, consider, for example, an aggregate surface with only acentral surface. Referring to FIG. 2A, the rays that are projected fromeither end of the display surface create an ideal holographic volume andideal viewing volumes as described above. Now consider, for example, ifthe central surface included two side surfaces angled towards theviewer. In this case, ray 256 and ray 257 would be projected at agreater angle from a normal of the central surface. Thus, the field ofview of the viewing volume would increase. Similarly, the holographicviewing volume would be nearer the display surface. Additionally,because the two second and third portions tilted nearer the viewingvolume, the holographic objects that are projected at a fixed distancefrom the display surface are closer to that viewing volume.

To simplify, a display surface with only a central surface has a planarfield of view, a planar threshold separation between the (central)display surface and the viewing volume, and a planar proximity between aholographic object and the viewing volume. Adding one or more sidesurfaces angled towards the viewer increases the field of view relativeto the planar field of view, decreases the separation between thedisplay surface and the viewing volume relative to the planarseparation, and increases the proximity between the display surface anda holographic object relative to the planar proximity. Further anglingthe side surfaces towards the viewer further increases the field ofview, decreases the separation, and increases the proximity. In otherwords, the angled placement of the side surfaces increases the immersiveexperience for viewers. Additionally, deflection optics may be used tooptimize the size and position of the viewing volume for LF displayparameters (e.g., dimensions and FOV).

Returning to FIG. 4D, in a similar example, FIG. 4D illustrates a sideview of a LF display system 450B with an aggregate surface in a “sloped”configuration. In this example, the LF display system 450B is located ina room with a front wall 452, a rear wall 454, a first sidewall (notshown), a second sidewall (not shown), a ceiling 472, and a floor 474.The first sidewall, the second sidewall, the rear wall 454, floor 474,and the ceiling 472 are all orthogonal. The LF display system 450Bincludes LF display modules tiled to form an aggregate surface 460covering the front wall. The front wall 452, and thus the aggregatesurface 460, includes three portions: (i) a first portion 462approximately parallel with the rear wall 454 (i.e., a central surface),(ii) a second portion 464 connecting the first portion 462 to theceiling 472 and angled to project energy towards the center of the room(i.e., a first side surface), and (iii) a third portion 464 connectingthe first portion 462 to the floor 474 and angled to project energytowards the center of the room (i.e., a second side surface). The firstportion is a vertical plane in the room and has a horizontal and avertical axis. The second and third portions are angled towards thecenter of the room along the vertical axis.

In this example, the viewing volume 468B of the LF display system 4508is in the center of the room and partially surrounded by the threeportions of the aggregate surface 460. Similar to the configurationshown in FIG. 4C, the two side portions (e.g., second portion 464, andthird portion 466) are angled to surround the viewer and form asurrounding surface. The surrounding surface increases the viewing FOVfrom the perspective of any viewer in the holographic viewing volume468B. Additionally, the surrounding surface allows the viewing volume468B to be closer to the surface of the displays such that projectedobjects appear closer. In other words, the angled placement of the sidesurfaces increases the field of view, decreases the separation, andincreases the proximity of the aggregate surface, thereby increasing theimmersive experience for viewers. Further, as will be discussed below,deflection optics may be used to optimize the size and position of theviewing volume 4688.

The sloped configuration of the side portions of the aggregate surface460 enables holographic content to be presented closer to the viewingvolume 468B than if the third portion 466 was not sloped. For example,the lower extremities (e.g., legs) of a character presented form a LFdisplay system in a sloped configuration may seem closer and morerealistic than if a LF display system with a flat front wall were used.

Additionally, the configuration of the LF display system and theenvironment which it is located may inform the shape and locations ofthe viewing volumes and viewing sub-volumes.

FIG. 4E, for example, illustrates a top down view of a LF display system450C with an aggregate surface 460 on a front wall 452 of a room. Inthis example, the LF display system 450D is located in a room with afront wall 452, a rear wall 454, a first sidewall 456, a second sidewall458, a ceiling (not shown), and a floor (not shown).

LF display system 450C projects various rays from the aggregate surface460. From each position on the display surface, light rays are projectedin an angular range that is centered on the viewing volume. The raysprojected from the left side of the aggregate surface 460 havehorizontal angular range 481, rays projected from the right side of theaggregate surface have horizontal angular range 482, and rays projectedfrom the center of the aggregate surface 460 have horizontal angularrange 483. In between these points, the projected rays may take onintermediate values of angle ranges. Having a gradient deflection anglein the projected rays across the display surface in this manner createsa viewing volume 468C. Further, this configuration avoids wastingresolution of the display on projecting rays into the side walls 456 and458.

FIG. 4F illustrates a side view of a LF display system 450D with anaggregate surface 460 on a front wall 452 of a room. In this example,the LF display system 450E is located in a room with a front wall 452, arear wall 454, a first sidewall (not shown), a second sidewall (notshown), a ceiling 472, and a floor 474. In this example, the floor istiered such that each tier rises in steps moving from the front wall tothe back wall. Here, each tier of the floor includes a viewingsub-volume (e.g., viewing sub volume 470A and 470B). A tiered floorallows for viewing sub-volumes that do not overlap. That is, eachviewing sub-volume has a line of sight from the viewing sub-volume tothe aggregate surface 460 that does not pass through another viewingsub-volume. In other words, this orientation produces a “stadiumseating” effect in which the vertical offset between tiers allows eachtier to “see over” the viewing sub-volumes of other tiers. LF displaysystems including viewing sub-volumes that do not overlap may provide ahigher quality viewing experience than LF display systems that haveviewing volumes that do overlap. For example, in the configuration shownin FIG. 4F, different holographic content may be projected to theaudiences in viewing sub-volumes 470A and 470B.

Control of a LF Display System

FIG. 5 is a block diagram of a LF display system 500, in accordance withone or more embodiments. The LF display system 500 comprises a LFdisplay assembly 510 and a controller 520. The LF display assembly 510includes one or more LF display modules 512 which project a light field.A LF display module 512 may include a source/sensor system 514 thatincludes an integrated energy source(s) and/or energy sensor(s) whichproject and/or sense other types of energy. The controller 520 includesa datastore 522, a network interface 524, and a LF processing engine530. The controller 520 may also include a tracking module 526, and aviewer profiling module 528. In some embodiments, the LF display system500 also includes a sensory feedback system 570 and a tracking system580. The LF display systems described in the context of FIGS. 1-4 areembodiments of the LF display system 500. In other embodiments, the LFdisplay system 500 comprises additional or fewer modules than thosedescribed herein. Similarly, the functions can be distributed among themodules and/or different entities in a different manner than isdescribed here. Applications of the LF display system 500 are alsodiscussed in detail below with regard to FIGS. 6-9.

The LF display assembly 510 provides holographic content in aholographic object volume that may be visible to viewers located withina viewing volume. The LF display assembly 510 may provide holographiccontent by executing display instructions received from the controller520. The holographic content may include one or more holographic objectsthat are projected in front of an aggregate surface the LF displayassembly 510, behind the aggregate surface of the LF display assembly510, or some combination thereof. Generating display instructions withthe controller 520 is described in more detail below.

The LF display assembly 510 provides holographic content (e.g., imagesof participants, avatars of participants, holographic objects, and/orother sensory content) using one or more LF display modules (e.g., anyof the LF display module 110, the LF display system 200, and LF displaymodule 300) included in an LF display assembly 510. For convenience, theone or more LF display modules may be described herein as LF displaymodule 512. The LF display module 512 can be tiled to form a LF displayassembly 510. The LF display modules 512 may be structured as variousseamless surface environments (e.g., single sided, multi-sided, a wallof a cinema, a curved surface, etc.). That is, the tiled LF displaymodules form an aggregate surface. As previously described, a LF displaymodule 512 includes an energy device layer (e.g., energy device layer220) and an energy waveguide layer (e.g., energy waveguide layer 240)that present holographic content. The LF display module 512 may alsoinclude an energy relay layer (e.g., energy relay layer 230) thattransfers energy between the energy device layer and the energywaveguide layer when presenting holographic content.

The LF display module 512 may also include other integrated systemsconfigured for energy projection and/or energy sensing as previouslydescribed. For example, a light field display module 512 may include anynumber of energy devices (e.g., energy device 340) configured to projectand/or sense energy. For convenience, the integrated energy projectionsystems and integrated energy sensing systems of the LF display module512 may be described herein, in aggregate, as the source/sensor system514. The source/sensor system 514 is integrated within the LF displaymodule 512, such that the source/sensor system 514 shares the sameseamless energy surface with LF display module 512. In other words, theaggregate surface of an LF display assembly 510 includes thefunctionality of both the LF display module 512 and the source/sensormodule 514. That is, an LF assembly 510 including a LF display module512 with a source/sensor system 514 may project energy and/or senseenergy while simultaneously projecting a light field. For example, theLF display assembly 510 may include a LF display module 512 andsource/sensor system 514 configured as a dual-energy surface orbidirectional energy surface as previously described.

In some embodiments, the LF display system 500 augments the generatedholographic content with other sensory content (e.g., coordinated touch,audio, or smell) using a sensory feedback system 570. The sensoryfeedback system 570 may augment the projection of holographic content byexecuting display instructions received from the controller 520.Generally, the sensory feedback system 570 includes any number ofsensory feedback devices external to the LF display assembly 510 (e.g.,sensory feedback system 442). Some example sensory feedback devices mayinclude coordinated acoustic projecting and receiving devices, aromaprojecting devices, temperature adjustment devices, force actuationdevices, pressure sensors, transducers, etc. In some cases, the sensoryfeedback system 570 may have similar functionality to the light fielddisplay assembly 510 and vice versa. For example, both a sensoryfeedback system 570 and a light field display assembly 510 may beconfigured to generate a sound field. As another example, the sensoryfeedback system 570 may be configured to generate haptic surfaces whilethe light field display 510 assembly is not.

To illustrate, in an example embodiment of a light field display system500, a sensory feedback system 570 may include one or more acousticprojection devices. The one or more acoustic projection devices areconfigured to generate one or more pressure waves that complement theholographic content when executing display instructions received fromthe controller 520. The generated pressure waves may be, e.g., audible(for sound), ultrasonic (for touch), or some combination thereof.Similarly, the sensory feedback system 570 may include an aromaprojecting device. The aroma projecting device may be configured toprovide scents to some, or all, of the target area when executingdisplay instructions received from the controller. The aroma devices maybe tied into an air circulation system (e.g., ducting, fans, or vents)to coordinate air flow within the target area. Further, the sensoryfeedback system 570 may include a temperature adjustment device. Thetemperature adjustment device is configured to increase or decreasetemperature in some, or all, of the target area when executing displayinstructions received from the controller 520.

In some embodiments, the sensory feedback system 570 is configured toreceive input from viewers of the LF display system 500. In this case,the sensory feedback system 570 includes various sensory feedbackdevices for receiving input from viewers. The sensor feedback devicesmay include devices such as acoustic receiving devices (e.g., amicrophone), pressure sensors, joysticks, motion detectors, transducers,etc. The sensory feedback system may transmit the detected input to thecontroller 520 to coordinate generating holographic content and/orsensory feedback.

To illustrate, in an example embodiment of a light field displayassembly 510, a sensory feedback system 570 includes a microphone. Themicrophone is configured to record audio produced by one or more viewers(e.g., participants in a video conference). The sensory feedback system570 provides the recorded audio to the controller 520 as viewer input.The controller 520 may use the viewer input to generate holographiccontent. For example, if a participant mentions a particular product, aholographic representation of the product may be generated to providecontext for the participant's comments. Similarly, the sensory feedbacksystem 570 may include a pressure sensor. The pressure sensor isconfigured to measure forces applied by viewers to the pressure sensor.The sensory feedback system 570 may provide the measured forces to thecontroller 520 as viewer input.

In some embodiments, the LF display system 500 includes a trackingsystem 580. The tracking system 580 includes any number of trackingdevices configured to determine the position, movement and/orcharacteristics of viewers in the target area. Generally, the trackingdevices are external to the LF display assembly 510. Some exampletracking devices include a camera assembly (“camera”), a depth sensor,structured light, a LIDAR system, a card scanning system, or any othertracking device that can track viewers within a target area.

The tracking system 580 may include one or more energy sources thatilluminate some or all of the target area with light. However, in somecases, the target area is illuminated with natural light and/or ambientlight from the LF display assembly 510 when presenting holographiccontent. The energy source projects light when executing instructionsreceived from the controller 520. The light may be, e.g., a structuredlight pattern, a pulse of light (e.g., an IR flash), or some combinationthereof. The tracking system may project light in the visible band (˜380nm to 750 nm), in the infrared (IR) band (˜750 nm to 1700 nm), in theultraviolet band (10 nm to 380 nm), some other portion of theelectromagnetic spectrum, or some combination thereof. A source mayinclude, e.g., a light emitted diode (LED), a micro LED, a laser diode,a TOF depth sensor, a tunable laser, etc.

The tracking system 580 may adjust one or more emission parameter whenexecuting instructions received from the controller 520. An emissionparameter is a parameter that affects how light is projected from asource of the tracking system 580. An emission parameter may include,e.g., brightness, pulse rate (to include continuous illumination),wavelength, pulse length, some other parameter that affects how light isprojected from the source assembly, or some combination thereof. In oneembodiment, a source projects pulses of light in a time-of-flightoperation.

The camera of the tracking system 580 captures images of the light(e.g., structured light pattern) reflected from the target area. Thecamera captures images when executing tracking instructions receivedfrom the controller 520. As previously described, the light may beprojected by a source of the tracking system 580. The camera may includeone or more cameras. That is, a camera may be, e.g., an array (1D or 2D)of photodiodes, a CCD sensor, a CMOS sensor, some other device thatdetects some or all of the light project by the tracking system 580, orsome combination thereof. In an embodiment, the tracking system 580 maycontain a light field camera external to the LF display assembly 510. Inother embodiments, the cameras are included as part of the LF displaysource/sensor module 514 included in the LF display assembly 510. Forexample, as previously described, if the energy relay element of alightfield module 512 is a bidirectional energy layer which interleaves bothemissive displays and imaging sensors at the energy device layer 220,the LE display assembly 510 can be configured to simultaneously projectlight fields and record imaging information from the viewing area infront of the display. In one embodiment, the captured images from thebidirectional energy surface form a light field camera. The cameraprovides captured images to the controller 520.

The camera of the tracking system 580 may adjust one or more imagingparameters when executing tracking instructions received from thecontroller 520. An imaging parameter is a parameter that affects how thecamera captures images. An imaging parameter may include, e.g., framerate, aperture, gain, exposure length, frame timing, rolling shutter orglobal shutter capture modes, some other parameter that affects how thecamera captures images, or some combination thereof.

The controller 520 controls the LF display assembly 510 and any othercomponents of the LF display system 500. The controller 520 comprises adata store 522, a network interface 524, a tracking module 526, a viewerprofiling module 528, and a light field processing engine 530. In otherembodiments, the controller 520 comprises additional or fewer modulesthan those described herein. Similarly, the functions can be distributedamong the modules and/or different entities in a different manner thanis described here. For example, the tracking module 526 may be part ofthe LF display assembly 510 or the tracking system 580.

The data store 522 is a memory that stores information for the LFdisplay system 500. The stored information may include displayinstructions, tracking instructions, emission parameters, imagingparameters, a virtual model of a target area, tracking information,images captured by the camera, one or more viewer profiles, calibrationdata for the light field display assembly 510, configuration data forthe LF display system 510 including resolution and orientation of LFmodules 512, desired viewing volume geometry, content for graphicscreation including 3D models, scenes and environments, materials andtextures, other information that may be used by the LF display system500, or some combination thereof. The data store 522 is a memory, suchas a read only memory (ROM), dynamic random access memory (DRAM), staticrandom access memory (SRAM), or some combination thereof.

The network interface 524 allows the light field display system tocommunicate with other systems or environments via a network. In oneexample, the LF display system 500 receives holographic content from aremote light field display system via the network interface 524. Inanother example, the LF display system 500 transmits holographic contentto a remote data store using the network interface 524.

The tracking module 526 tracks viewers viewing content presented by theLF display system 500. To do so, the tracking module 526 generatestracking instructions that control operation of the source(s) and/or thecamera(s) of the tracking system 580, and provides the trackinginstructions to the tracking system 580. The tracking system 580executes the tracking instructions and provides tracking input to thetracking module 526.

The tracking module 526 may determine a position of one or more viewerswithin the target area (e.g., sitting in a particular chair in aconference room, walking around the conference room, etc.). Thedetermined position may be relative to, e.g., some reference point(e.g., a display surface, a conference table, etc.). In otherembodiments, the determined position may be within the virtual model ofthe target area. The tracked position may be, e.g., the tracked positionof a viewer and/or a tracked position of a portion of a viewer (e.g.,eye location, hand location, etc.). The tracking module 526 determinesthe position using one or more captured images from the cameras of thetracking system 580. The cameras of the tracking system 580 may bedistributed about the LF display system 500, and can capture images instereo, allowing for the tracking module 526 to passively track viewers.In other embodiments, the tracking module 526 actively tracks viewers.That is, the tracking system 580 illuminates some portion of the targetarea, images the target area, and the tracking module 526 uses time offlight and/or structured light depth determination techniques todetermine position. The tracking module 526 generates trackinginformation using the determined positions.

The tracking module 526 may also receive tracking information as inputsfrom viewers of the LF display system 500. The tracking information mayinclude body movements that correspond to various input options that theviewer is provided by the LF display system 500. For example, thetracking module 526 may track a viewer's body movement and assign anyvarious movement as an input to the LF processing engine 530. Thetracking module 526 may provide the tracking information to the datastore 522, the LF processing engine 530, the viewer profiling module528, any other component of the LF display system 500, or somecombination thereof.

To provide context for the tracking module 526, consider an exampleembodiment of an LF display system 500 provides video conferencing for ameeting. In response to one participant calling a vote on a proposal,one or more participants raise their hands. The tracking system 580 mayrecord the movement of the participants' hands and transmit therecording to the tracking module 526. The tracking module 526 tracks themotion of the participants' hands in the recording and sends the inputto LF processing engine 530. The viewer profiling module 528, asdescribed below, determines whether information in the image indicatesthat motion of the participants' hands is associated with a vote infavor of the proposal. The LF processing engine 530 may generateappropriate holographic content indicating the result of the vote. Forexample, the LF processing engine 530 may project a tally of the votes.

The LF display system 500 includes a viewer profiling module 528configured to identify and profile viewers. The viewer profiling module528 generates a profile of a viewer (or viewers) that views holographiccontent displayed by a LF display system 500. The viewer profilingmodule 528 generates a viewer profile based, in part, on viewer inputand monitored viewer behavior, actions, and reactions. The viewerprofiling module 528 can access information obtained from trackingsystem 580 (e.g., recorded images, videos, sound, etc.) and process thatinformation to determine various information. In various examples,viewer profiling module 528 can use any number of machine vision ormachine hearing algorithms to determine viewer behavior, actions, andreactions. Monitored viewer behavior can include, for example, smiles,raising a hand, cheering, clapping, laughing, and/or other changes ingestures, or movement by the viewers, etc.

More generally, a viewer profile may include any information receivedand/or determined about a viewer viewing holographic content from the LFdisplay system. For example, each viewer profile may log actions orresponses of that viewer to the content displayed by the LF displaysystem 500. Some example information that can be included in a viewerprofile are provided below.

In some embodiments, a viewer profile can indicate a role of the viewerin relation to an organization or group associated with the LF displaysystem 500. For example, in a video conferencing system operated by abusiness, a viewer profile might include, for example, the viewer's jobtitle, responsibilities, authorization to view confidential information,etc. A viewer profile may additionally or alternatively indicate moregeneral viewer characteristics such as, for example, age, sex,ethnicity, clothing, location, etc.

In some embodiments, a viewer profile can indicate viewer preferencesregarding the presentation of holographic content. For example, a viewerprofile may indicate holographic object volumes to display holographiccontent (e.g., to the viewer's right) and/or holographic object volumesto not display holographic content (e.g., to the viewer's left). Theviewer profile may also indicate that the viewer prefers to have hapticinterfaces presented near them, or prefers to avoid them.

In some embodiments, a viewer profile may also describe characteristicsand preferences for a group of viewers rather than a particular viewer.For example, viewer profiling module 528 may generate a viewer profilefor particular combinations of viewers in a video conference. In oneexample, viewer profiling module 528 creates a profile for a pair ofviewers indicating that, when the pair engage in a video conferencesession using a LF display system 500, they prefer a particularconfiguration of holographic viewing zones, haptic interfaces,background images, and/or avatars, etc. Any of the previously describedinformation and characteristics may be applied to a group of viewers.

The viewer profiling module 528 may also access a profile associatedwith a particular viewer (or viewers) from a third-party system orsystems to build a viewer profile. For example, a viewer may connect oneor more social media accounts to the viewer's profile maintained by theviewer profiling module 528. The viewer profiling module 528 may accessinformation from one or more of the social media accounts to build (oraugment) the viewer's profile.

In some embodiments, the data store 522 includes a viewer profile storethat stores viewer profiles generated, updated, and/or maintained by theviewer profiling module 528. The viewer profile can be updated in thedata store at any time by the viewer profiling module 528. For example,in an embodiment, the viewer profile store receives and storesinformation regarding a particular viewer in their viewer profile whenthe particular viewer views holographic content provided by the LFdisplay system 500. In this example, the viewer profiling module 528includes a facial recognition algorithm that may recognize viewers andpositively identify them as they view presented holographic content. Toillustrate, as a viewer enters the target area of the LF display system500, the tracking system 580 obtains an image of the viewer. The viewerprofiling module 528 inputs the captured image and identifies theviewer's face using the facial recognition algorithm. The identifiedface is associated with a viewer profile in the profile store and, assuch, all input information obtained about that viewer may be stored intheir profile. The viewer profiling module 528 may also utilize cardidentification scanners, voice identifiers, a radio-frequencyidentification (RFID) chip scanners, barcode scanners, etc. topositively identify a viewer.

In embodiments where the viewer profiling module 528 can positivelyidentify viewers, the viewer profiling module 528 can determine eachvisit of each viewer to the LF display system 500. The viewer profilingmodule 528 may then store the time and date of each visit in the viewerprofile for each viewer. Similarly, the viewer profiling module 528 maystore received inputs from a viewer from any combination of the sensoryfeedback system 570, the tracking system 580, and/or the LF displayassembly 510 each time they occur. The viewer profile system 528 mayadditionally receive further information about a viewer from othermodules or components of the controller 520 which can then be storedwith the viewer profile. Other components of the controller 520 may thenalso access the stored viewer profiles for determining subsequentcontent to be provided to that viewer.

The LF processing engine 530 generates holographic content comprised oflight field data, as well as data for all of the sensory domainssupported by a LF display system 500. For example, LF processing engine530 may generate 4D coordinates in a rasterized format (“rasterizeddata”) that, when executed by the LF display assembly 510, cause the LFdisplay assembly 510 to present holographic content. The LF processingengine 530 may access the rasterized data from the data store 522.Additionally, the LF processing engine 530 may construct rasterized datafrom a vectorized data set. Vectorized data is described below. The LFprocessing engine 530 can also generate sensory instructions required toprovide sensory content that augments the holographic objects. Asdescribed above, sensory instructions may generate, when executed by theLF display system 500, haptic surfaces, sound fields, and other forms ofsensory energy supported by the LF display system 500. The LF processingengine 530 may access sensory instructions from the data store 522, orconstruct the sensory instructions form a vectorized data set. Inaggregate, the 4D coordinates and sensory data represent holographicdata as display instructions executable by a LF display system togenerate holographic and sensory content. More generally, holographiccontent can take the form of computer graphics (CG) content with ideallight field coordinates, live action content, rasterized data,vectorized data, electromagnetic energy transported by a set of relays,instructions sent to a group of energy devices, energy locations on oneor more energy surfaces, the set of energy propagation paths that areprojected from the display surface, a holographic object that is visibleto a viewer or an audience, and many other similar forms.

The amount of rasterized data describing the flow of energy through thevarious energy sources in a LF display system 500 is incredibly large.While it is possible to display the rasterized data on a LF displaysystem 500 when accessed from a data store 522, it is untenable toefficiently transmit, receive (e.g., via a network interface 524), andsubsequently display the rasterized data on a LF display system 500.Take, for example, rasterized data representing a short film forholographic projection by a LF display system 500. In this example, theLF display system 500 includes a display containing several gigapixelsand the rasterized data contains information for each pixel location onthe display. The corresponding size of the rasterized data is vast(e.g., many gigabytes per second of film display time), and unmanageablefor efficient transfer over commercial networks via a network interface524. The efficient transfer problem may be amplified for applicationsincluding live streaming of holographic content. An additional problemwith merely storing rasterized data on data store 522 arises when aninteractive experience is desired using inputs from the sensory feedbacksystem 570 or the tracking module 526. To enable an interactiveexperience, the light field content generated by the LF processingengine 530 can be modified in real-time in response to sensory ortracking inputs. In other words, in some cases, LF content cannot simplybe read from the data store 522.

Therefore, in some configurations, data representing holographic contentfor display by a LF display system 500 may be transferred to the LFprocessing engine 530 in a vectorized data format (“vectorized data”).Vectorized data may be orders of magnitude smaller than rasterized data.Further, vectorized data provides high image quality while having a dataset size that enables efficient sharing of the data. For example,vectorized data may be a sparse data set derived from a denser data set.Thus, vectorized data may have an adjustable balance between imagequality and data transmission size based on how sparse vectorized datais sampled from dense rasterized data. Tunable sampling to generatevectorized data enables optimization of image quality for a givennetwork speed. Consequently, vectorized data enables efficienttransmission of holographic content via a network interface 524.Vectorized data also enables holographic content to be live-streamedover a commercial network.

In summary, the LF processing engine 530 may generate holographiccontent derived from rasterized data accessed from the data store 522,vectorized data accessed from the data store 522, or vectorized datareceived via the network interface 524. In various configurations,vectorized data may be encoded before data transmission and decodedafter reception by the LF controller 520. In some examples, thevectorized data is encoded for added data security and performanceimprovements related to data compression. For example, vectorized datareceived by the network interface may be encoded vectorized datareceived from a holographic streaming application. In some examples,vectorized data may require a decoder, the LF processing engine 530, orboth of these to access information content encoded in vectorized data.The encoder and/or decoder systems may be available to customers orlicensed to third-party vendors.

Vectorized data contains all the information for each of the sensorydomains supported by a LF display system 500 in a way that may supportan interactive experience. For example, vectorized data for aninteractive holographic experience may include any vectorized propertiesthat can provide accurate physics for each of the sensory domainssupported by a LF display system 500. Vectorized properties may includeany properties that can be synthetically programmed, captured,computationally assessed, etc. A LF processing engine 530 may beconfigured to translate vectorized properties in vectorized data torasterized data. The LF processing engine 530 may then projectholographic content translated from the vectorized data using the LFdisplay assembly 510. In various configurations, the vectorizedproperties may include one or more red/green/blue/alpha channel(RGBA)+depth images, multi view images with or without depth informationat varying resolutions that may include one high-resolution center imageand other views at a lower resolution, material properties such asalbedo and reflectance, surface normals, other optical effects, surfaceidentification, geometrical object coordinates, virtual cameracoordinates, display plane locations, lighting coordinates, tactilestiffness for surfaces, tactile ductility, tactile strength, amplitudeand coordinates of sound fields, environmental conditions, somatosensoryenergy vectors related to the mechanoreceptors for textures ortemperature, audio, and any other sensory domain property. Many othervectorized properties are also possible.

The LF display system 500 may also generate an interactive viewingexperience. That is, holographic content may be responsive to inputstimuli containing information about viewer locations, gestures,interactions, interactions with holographic content, or otherinformation derived from the viewer profiling module 528, and/ortracking module 526. For example, in an embodiment, a LF processingsystem 500 creates an interactive viewing experience using vectorizeddata of a real-time performance received via a network interface 524. Inanother example, if a holographic object needs to move in a certaindirection immediately in response to a viewer interaction, the LFprocessing engine 530 may update the render of the scene so theholographic object moves in that required direction. This may requirethe LF processing engine 530 to use a vectorized data set to renderlight fields in real time based a 3D graphical scene with the properobject placement and movement, collision detection, occlusion, color,shading, lighting, etc., correctly responding to the viewer interaction.The LF processing engine 530 converts the vectorized data intorasterized data for presentation by the LF display assembly 510.

The rasterized data includes holographic content instructions andsensory instructions (display instructions) representing the real-timeperformance. The LF display assembly 510 simultaneously projectsholographic and sensory content of the real-time performance byexecuting the display instructions. The LF display system 500 monitorsviewer interactions (e.g., vocal response, touching, etc.) with thepresented real-time performance with the tracking module 526 and viewerprofiling module 528. In response to the viewer interactions, the LFprocessing engine may create an interactive experience by generatingadditional holographic and/or sensory content for display to theviewers.

To illustrate, consider an example embodiment of an LF display system500 including a LF processing engine 530 that generates a holographicobject representing a product prototype. A viewer may move to touch theholographic object representing product prototype. Correspondingly, thetracking system 580 tracks movement of the viewer's hands relative tothe holographic object. The movement of the viewer is recorded by thetracking system 580 and sent to the controller 520. The tracking module526 continuously determines the motion of the viewer's hand and sendsthe determined motions to the LF processing engine 530. The LFprocessing engine 530 determines the placement of the viewer's hand inthe scene, adjusts the real-time rendering of the graphics to includeany required change in the holographic object (such as position, color,or occlusion). The LF processing engine 530 instructs the LF displayassembly 510 (and/or sensory feedback system 570) to generate a tactilesurface using the volumetric haptic projection system (e.g., usingultrasonic speakers). The generated tactile surface corresponds to atleast a portion of the holographic object and occupies substantially thesame space as some or all of an exterior surface of the holographicobject. The LF processing engine 530 uses the tracking information todynamically instruct the LF display assembly 510 to move the location ofthe tactile surface along with a location of the rendered holographicobject such that the viewer is given both a visual and tactileperception of touching the prototype. More simply, when a viewer viewshis hand touching a holographic prototype, the viewer simultaneouslyfeels haptic feedback indicating their hand touches the holographicprototype, and the prototype changes position or motion in response tothe touch. In some examples, rather than presenting an interactiveprototype accessed from a data store 522, the interactive prototype maybe received as part of holographic content received from alive-streaming application via a network interface 524 (e.g., theholographic prototype may be a holographic representation of a physicalprototype at a different physical location that is being imaged by adifferent LF display system 500.

In embodiments where the LF display system is used to provide videoconferencing, the holographic content may include holographic images ofone or more participants in the video conference. The holographiccontent may also include other holographic objects such as, for example,holographic handouts, holographic whiteboards, holographic movies orvideos, holographic simulations, holographic product prototypes,holographic models, holographic experiences, holographic games,holographic items, holographic assistants, any other holographic object,or any combination thereof. In some embodiments, holographic content maybe received from third party systems separate from the LF display system500.

In a video conferencing configuration, the participants are located intwo or more physical locations. At least one of the physical locationshas a LF capture system. The LF capture system can be a plenoptic lightfield camera, or a multiview camera system with multiple lenses andsensors, external to the LF display. Alternatively, the LF capturesystem can be integrated into the LF display assembly 510, as abidirectional energy surface which both projects LF as well as absorbsincident light, and relays the incident light to imaging sensors.

The data set generated in capturing the full LF may be unmanageable formost processors as well as efficient network transmission. To addressthis, the data recorded from the light field capture system may becompressed. In some embodiments, the data is reduced into a vectorizedformat including, for example, any of: N red/green/blue/alpha channel(RGBA)+depth images, N multi view images with or without depth atvarying resolutions that may include one high-resolution center imageand other views at a lower resolution, or any other reduced data set. Inthis way, the amount of data needed to represent the full LF may bereduced significantly (e.g., by multiple orders of magnitude), creatinga viable path to enabling the transmission of truly holographicdatasets. Such techniques can provide high image quality while balancingdata set size for efficient sharing of data providing advantagesincluding, for example, reduced storage requirements, enabling data tobe streamed live over a network, etc.

In some embodiments, the data is encoded by a proprietary encodingblock. This encoding process is part of a proprietary encoding/decodingpair. The receiving system or systems include a matching decoding block.These encoding and decoding blocks can be licensed by third partyvendors. The encoding block can compress the vectorized format whilebalancing image quality with transmission speed, and automaticallyadjust to network speeds, e.g., to provide the highest possible imagequality for the available bandwidth. The encoding process may useselectable or variable compression ratios which may include real-time oroff-line image processing, other computations, and/or reducing the datato a sparser data set.

The encoded data is sent with a steaming engine through a networkinterface (e.g., network interface 524, in the case where the LF capturesystem is integrated with a LF display system 500) from the LF capturesystem to one or more LF display systems 500 (e.g., located in thephysical location or locations of the other participant orparticipants). The encoded data is received by a LF display system 500,decoded (e.g., by the proprietary decoder). The decoder processes theencoded signal, and combined with the display drivers and displayhardware configuration, allows the LF processing engine 530 toagnostically project the available information as a fully rasterized 4Dlight field, taking into account the resolution of the LF displayassembly 510 as well as the available haptic interfaces and othersensory projection capabilities of the LF display system 500.

As described previously, the encoded data sent over the network is in avectorized format. The LF processing engine 530 takes this data andconverts it into a rasterized format that drives the LF display assembly510. The rasterized format may be many orders of magnitude larger thanthe vectorized data set. In some embodiments, both holographic videoconferencing information and CG content is sent.

In some embodiments, the encoded data also includes matching hapticsurface instructions for some or all of the holographic objects. In oneembodiment, the vectorized format produced from the light field displayassembly 510 includes vectorized properties that provide accuratephysics for multiple sensory domains for which properties may besynthetically programmed, captured, or computationally assessed,including, for example, any of: N red/green/blue/alpha channel(RGBA)+depth images, N multi view images with or without depth atvarying resolutions that may include one high-resolution center imageand other views at a lower resolution, material properties such asalbedo and reflectance, surface normals, other optical effects, surfaceidentification, geometrical object coordinates, virtual cameracoordinates, display plane locations, lighting coordinates, tactilestiffness for surfaces, tactile ductility, tactile strength, amplitudeand coordinates of sound fields, environmental conditions, somatosensoryenergy vectors related to the mechanoreceptors for textures ortemperature, game audio, and any other sensory domain property. Thevectorization of data may eliminate multiple orders of magnitude ofrequired data, creating a viable path to enabling the transmission ofdatasets including properties in multiple sensory domains.

The LF processing engine 500 may also modify holographic content to suitthe space in which the holographic content is being presented. Forexample, not every conference room is the same size, has the same numberof seats, or has the same technical configuration. As such, LFprocessing engine 530 may modify holographic content such that it willbe appropriately displayed in a conference room. In one embodiment, theLF processing engine 530 may access a configuration file of a conferenceroom including the layout, resolution, field-of-view, other technicalspecifications, etc. of the conference room. The LF processing engine530 may render and present the holographic content based on informationincluded in the configuration file.

The LF processing engine 530 may also create holographic content fordisplay by the LF display system 500. Importantly, here, creatingholographic content for display is different from accessing, orreceiving, holographic content for display. That is, when creatingcontent, the LF processing engine 530 generates entirely new content fordisplay rather than accessing previously generated and/or receivedcontent. The LF processing engine 530 can use information from thetracking system 580, the sensory feedback system 570, the viewerprofiling module 528, the tracking module 526, or some combinationthereof, to create holographic content for display. In some examples, LFprocessing engine 530 may access information from elements of the LFdisplay system 500 (e.g., tracking information and/or a viewer profile),create holographic content based on that information, and display thecreated holographic content using the LF display system 500 in response.The created holographic content may be augmented with other sensorycontent (e.g., touch, audio, or smell) when displayed by the LF displaysystem 500.

Dynamic Content Generation for a LF Display System

In some embodiments, the LF processing engine 530 incorporates anartificial intelligence (A) model to create holographic content fordisplay by the LF display system 500. The AI model may includesupervised or unsupervised learning algorithms including but not limitedto regression models, neural networks, classifiers, or any other AIalgorithm. The AI model may be used to determine viewer preferencesbased on viewer information recorded by the LF display system 500 (e.g.,by tracking system 580) which may include information on a viewer'sbehavior.

The AI model may access information from the data store 522 to createholographic content. For example, the AI model may access viewerinformation from a viewer profile or profiles in the data store 522 ormay receive viewer information from the various components of the LFdisplay system 500. To illustrate, the AI model may determine one viewerprefers seeing financial data in a chart while another prefers seeing itin a table. The AI model may determine the preference based on theviewers' reactions or responses to previously viewed holographic contentincluding financial data (e.g., Did the viewer ask for clarification?Did the user ask to see the data in a different format? Etc.). The LFdisplay system 500 may then present the same data to different usersusing different types of holographic representation. That is, the AImodel may create holographic content personalized to a set of viewersaccording to the learned preferences of those viewers. So, for example,the AI model may create a holographic chart of financial data for oneuser and a holographic table of the same data for another. The AI modelmay also store the learned preferences of each viewer in the viewerprofile store of the data store 522. In some examples, the AI model maycreate holographic content tailored to a group of viewers rather than asingle viewer.

One example of an AI model that can be used to identify characteristicsof viewers, identify reactions, and/or generate holographic contentbased on the identified information is a convolutional neural networkmodel with layers of nodes, in which values at nodes of a current layerare a transformation of values at nodes of a previous layer. Atransformation in the model is determined through a set of weights andparameters connecting the current layer and the previous layer. Forexample, and AI model may include five layers of nodes: layers A, B, C,D, and E. The transformation from layer A to layer B is given by afunction W₁, the transformation from layer B to layer C is given by W₂,the transformation from layer C to layer D is given by W₁, and thetransformation from layer D to layer E is given by W₄. In some examples,the transformation can also be determined through a set of weights andparameters used to transform between previous layers in the model. Forexample, the transformation W₄ from layer D to layer E can be based onparameters used to accomplish the transformation W₁ from layer A to B.

The input to the model can be an image taken by tracking system 580encoded onto the convolutional layer A and the output of the model isholographic content decoded from the output layer E. Alternatively oradditionally, the output may be a determined characteristic of a viewerin the image. In this example, the AI model identifies latentinformation in the image representing viewer characteristics in theidentification layer C. The AI model reduces the dimensionality of theconvolutional layer A to that of the identification layer C to identifyany characteristics, actions, responses, etc. in the image. In someexamples, the AI model then increases the dimensionality of theidentification layer C to generate holographic content.

The image from the tracking system 580 is encoded to a convolutionallayer A. Images input in the convolutional layer A can be related tovarious characteristics and/or reaction information, etc. in theidentification layer C. Relevance information between these elements canbe retrieved by applying a set of transformations between thecorresponding layers. That is, a convolutional layer A of an AI modelrepresents an encoded image, and identification layer C of the modelrepresents a smiling viewer. Smiling viewers in a given image may beidentified by applying the transformations W₁ and W₂ to the pixel valuesof the image in the space of convolutional layer A. The weights andparameters for the transformations may indicate relationships betweeninformation contained in the image and the identification of a smilingviewer. For example, the weights and parameters can be a quantization ofshapes, colors, sizes, etc. included in information representing asmiling viewer in an image. The weights and parameters may be based onhistorical data (e.g., previously tracked viewers).

Smiling viewers in the image are identified in the identification layerC. The identification layer C represents identified smiling viewersbased on the latent information about smiling viewers in the image.

Identified smiling viewers in an image can be used to generateholographic content. To generate holographic content, the AI modelstarts at the identification layer C and applies the transformations W₂and W₃ to the value of the given identified smiling viewers in theidentification layer C. The transformations result in a set of nodes inthe output layer E. The weights and parameters for the transformationsmay indicate relationships between an identified smiling viewers andspecific holographic content and/or preferences. In some cases, theholographic content is directly output from the nodes of the outputlayer E, while in other cases the content generation system decodes thenodes of the output layer E into a holographic content. For example, ifthe output is a set of identified characteristics, the LF processingengine can use the characteristics to generate holographic content.

Additionally, the AI model can include layers known as intermediatelayers. Intermediate layers are those that do not correspond to animage, identifying characteristics/reactions, etc., or generatingholographic content. For example, in the given example, layer B is anintermediate layer between the convolutional layer A and theidentification layer C. Layer D is an intermediate layer between theidentification layer C and the output layer E. Hidden layers are latentrepresentations of different aspects of identification that are notobserved in the data, but may govern the relationships between theelements of an image when identifying characteristics and generatingholographic content. For example, a node in the hidden layer may havestrong connections (e.g., large weight values) to input values andidentification values that share the commonality of “laughing peoplesmile.” As another example, another node in the hidden layer may havestrong connections to input values and identification values that sharethe commonality of “scared people scream.” Of course, any number oflinkages are present in a neural network. Additionally, eachintermediate layer is a combination of functions such as, for example,residual blocks, convolutional layers, pooling operations, skipconnections, concatenations, etc. Any number of intermediate layers Bcan function to reduce the convolutional layer to the identificationlayer and any number of intermediate layers D can function to increasethe identification layer to the output layer.

In one embodiment, the AI model includes deterministic methods that havebeen trained with reinforcement learning (thereby creating areinforcement learning model). The model is trained to increase thequality of the performance using measurements from tracking system 580as inputs, and changes to the created holographic content as outputs.

Reinforcement learning is a machine learning system in which a machinelearns ‘what to do’—how to map situations to actions—so as to maximize anumerical reward signal. The learner (e.g. LF processing engine 530) isnot told which actions to take (e.g., generating prescribed holographiccontent), but instead discovers which actions yield the most reward(e.g., increasing the quality of holographic content by making morepeople cheer) by trying them. In some cases, actions may affect not onlythe immediate reward but also the next situation and, through that, allsubsequent rewards. These two characteristics—trial-and-error search anddelayed reward—are two distinguishing features of reinforcementlearning.

Reinforcement learning is defined not by characterizing learningmethods, but by characterizing a learning problem. Basically, areinforcement learning system captures those important aspects of theproblem facing a learning agent interacting with its environment toachieve a goal. That is, in the example of generating a song for aperformer, the reinforcement learning system captures information aboutviewers in the venue (e.g., age, disposition, etc.). Such an agentsenses the state of the environment and takes actions that affect thestate to achieve a goal or goals (e.g., creating a pop song for whichthe viewers will cheer). In its most basic form, the formulation ofreinforcement learning includes three aspects for the learner:sensation, action, and goal. Continuing with the song example, the LFprocessing engine 530 senses the state of the environment with sensorsof the tracking system 580, displays holographic content to the viewersin the environment, and achieves a goal that is a measure of theviewer's reception of that song.

One of the challenges that arises in reinforcement learning is thetrade-off between exploration and exploitation. To increase the rewardin the system, a reinforcement learning agent prefers actions that ithas tried in the past and found to be effective in producing reward.However, to discover actions that produce reward, the learning agentselects actions that it has not selected before. The agent ‘exploits’information that it already knows in order to obtain a reward, but italso ‘explores’ information in order to make better action selections inthe future. The learning agent tries a variety of actions andprogressively favors those that appear to be best while still attemptingnew actions. On a stochastic task, each action is generally tried manytimes to gain a reliable estimate of its expected reward. For example,if the LF processing engine creates holographic content that the LFprocessing engine knows will result in a viewer laughing after a longperiod of time, the LF processing engine may change the holographiccontent such that the time until a viewer laughs decreases.

Further, reinforcement learning considers the whole problem of agoal-directed agent interacting with an uncertain environment.Reinforcement learning agents have explicit goals, can sense aspects oftheir environments, and can choose actions to receive high rewards(i.e., a roaring crowd). Moreover, agents generally operate despitesignificant uncertainty about the environment they face. Whenreinforcement learning involves planning, the system addresses theinterplay between planning and real-time action selection, as well asthe question of how environmental elements are acquired and improved.For reinforcement learning to make progress, important sub problems haveto be isolated and studied, the sub problems playing clear roles incomplete, interactive, goal-seeking agents.

The reinforcement learning problem is a framing of a machine learningproblem where interactions are processed and actions are carried out toachieve a goal. The learner and decision-maker is called the agent(e.g., LF processing engine 530). The thing it interacts with,comprising everything outside the agent, is called the environment(e.g., viewers in a venue, etc.). These two interact continually, theagent selecting actions (e.g., creating holographic content) and theenvironment responding to those actions and presenting new situations tothe agent. The environment also gives rise to rewards, special numericalvalues that the agent tries to maximize over time. In one context, therewards act to maximize viewer positive reactions to holographiccontent. A complete specification of an environment defines a task whichis one instance of the reinforcement learning problem.

To provide more context, an agent (e.g., LF processing engine 530) andenvironment interact at each of a sequence of discrete time steps, i.e.t=0, 1, 2, 3, etc. At each time step t the agent receives somerepresentation of the environment's state s_(t) (e.g., measurements fromtracking system 580). The states s_(t) are within S, where S is the setof possible states. Based on the state stand the time step t, the agentselects an action at (e.g., making the performer do the splits). Theaction at is within A(s_(t)), where A(s_(t)) is the set of possibleactions. One time state later, in part as a consequence of its action,the agent receives a numerical reward r_(t+1). The states r_(t+1) arewithin R, where R is the set of possible rewards. Once the agentreceives the reward, the agent selects in a new state s_(t+1).

At each time step, the agent implements a mapping from states toprobabilities of selecting each possible action. This mapping is calledthe agent's policy and is denoted π_(t) where π_(t)(s,a) is theprobability that a_(t)=a if s_(t)=s. Reinforcement learning methods candictate how the agent changes its policy as a result of the states andrewards resulting from agent actions. The agent's goal is to maximizethe total amount of reward it receives over time.

This reinforcement learning framework is flexible and can be applied tomany different problems in many different ways (e.g. generatingholographic content). The framework proposes that whatever the detailsof the sensory, memory, and control apparatus, any problem (orobjective) of learning goal-directed behavior can be reduced to threesignals passing back and forth between an agent and its environment: onesignal to represent the choices made by the agent (the actions), onesignal to represent the basis on which the choices are made (thestates), and one signal to define the agent's goal (the rewards).

Of course, the AI model can include any number of machine learningalgorithms. Some other AI models that can be employed are linear and/orlogistic regression, classification and regression trees, k-meansclustering, vector quantization, etc. Whatever the case, generally, theLF processing engine 530 takes an input from the tracking module 526and/or viewer profiling module 528 and a machine learning model createsholographic content in response. Similarly, the AI model may direct therendering of holographic content.

In some embodiments, the LF processing engine 530 generates displayinstructions for presentation of holographic assistants to a participantin video conference. Such an assistant may provide a participant withinformation pertinent to the video conference, assist withpresentations, and the like. The LF processing engine 530 may retrievethe holographic assistants stored as holographic objects in a datastore. Each holographic assistant may have various parameters thatdictate presentation of the holographic assistant. For example, theholographic assistant may have parameters that include but are notlimited to a type of assistant (e.g., human avatar, alien, robot,humanoid, etc.), a size of the assistant, a gender of the assistant whenavailable, a voice of the assistant, a personality of the assistant, orany combination thereof.

In additional embodiments, the LF processing engine 530 further accessesa viewer profile that may have preferences of a participant for anassistant. For example, the viewer profile may include preferences(e.g., provided by the participant or inferred by the viewer profilingmodule 528) that a participant prefers to have an assistant as a malehuman with brown hair and a deep voice, a female alien with black hairand a high-pitched voice, etc. Additionally, the LF processing engine530 may incorporate an AI model, as described above, to generateinstructions for presentation of the assistant so as to present aholographic assistant that engages with the participant. The AI modelmay be used to constantly generate instructions for the sensory feedbacksystem 570 to provide audio feedback to respond to voice input from theparticipant, i.e., simulating a real time dialog between the participantand the holographic assistant. Furthermore, the LF processing engine 530may use tracking information from the tracking system 580 and/or thetracking module 526 to generate display instructions for the LF displayassembly 510 to adjust presentation of the holographic assistant's eyesto move or track with the participant's gaze or body movement.

The preceding examples of creating content are not limiting. Mostbroadly, LF processing engine 530 creates holographic content fordisplay to viewers of a LF display system 500. The holographic contentcan be created based on any of the information included in the LFdisplay system 500.

Light Field Display Video Conferencing System

FIG. 6 is an illustration of a LF display system 600 for videoconferencing, in accordance with one or more embodiments. The LF displaysystem 600 is an embodiment of the LF display system 500. The LF displaysystem 600 is located in a conference space 610. The conference space610 is typically a room including a LF display formed by LF displaymodules 620 of a LF display assembly 510. In the configuration shown inFIG. 6, the LF display modules 620 are set up covering one wall of avideo conferencing room. However, the conference space 610 may be anyphysical space in which the LF display modules 620 may be temporarily orpermanently installed. In one embodiment, the LF display system 600 hasan imaging system that is integrated into the LF display assembly 510.Thus, the LF display assembly 510 may act as a bidirectional displaysurface that both projects light fields and relays light from thedisplay surface to at least one imaging sensor. The LF display systemmay additionally or alternatively have cameras external to the display,with the images analyzed using tracking software in a controller 520 forthe light field display assembly 510.

In FIG. 6, the conference space 610 also includes a table 630 and achair 640. The table 640 is positioned against the wall with the LFdisplay modules 620 and the chair 640 is positioned for a participant tosit at the table 630. The LF display modules 620 are generatingholographic images of a table 632 and chair 642 that are located in adifferent physical space 612 in which another video conferenceparticipant is located. The holographic images may be generated based onvisual data collected by a remote light field display assembly (e.g.,one or more remote LF display modules) located in a remote conferencespace 612. For example, in one embodiment, each conference spaceincludes an LF display system 600 that has an imaging system integratedinto a LF display assembly 510. Thus, each display assembly 510 acts asa bidirectional surface, which may be configured to give the impressionthat the two conference spaces are physically adjacent to each otherwith the LF display assemblies 510 serving as a window between them.

The visual data used to generate holographic images may be a digitalrepresentation of light detected by one or more imaging sensors (e.g.,imaging sensors of one or more LF display modules 620). For example, inthe case of the chair 642, the visual data may be light that wasreflected from the chair and detected by the LF display modules 620 ofan LF display system 600 in the remote conference space 612.Alternatively or additionally, the remote light field display assembly510 may collect visual data using other sensors, such as stand-alonecameras, etc. In some embodiments, the visual data may be vectorizedand/or compressed as described previously, with regard to FIG. 5.

The holographic table 632 is abutting the physical table 630. Thus, itappears to a participant in the conference space 610 (e.g., sitting inthe chair 640) that the physical chair 640 and the image of the chair642 in the other physical location 612 are around a single table (madeup of the physical table 630 and the image of the table 632 in the otherlocation). Similarly, in the other physical location 612, a holographictable 632 is presented abutting the physical instance of table 630 alongwith a holographic chair 642. Although FIG. 6 shows two identical tablesbeing used to improve the illusion that they are two halves of a singletable, different sizes and shapes of table may be used.

When participants sit in the chairs 640, 642 (or otherwise positionthemselves in the respective viewing volumes), holograms of theparticipants may be provided in the other physical location. Forexample, a hologram of a first user sitting in chair 640 may be providedin the second conference space 612 and a hologram of a secondparticipant sitting in chair 642 may be provided in the first conferencespace 610. Although each physical location is described as having asingle chair and participant, it should be appreciated that either orboth locations may include multiple participants. It should also beappreciated that in some embodiments, there may be more than twophysical locations.

In one embodiment, the remote conference space 612 contains no LFdisplay system, but is photographed with a LF camera, and the images aresent to the LF display system 600 in conference space 610. In otherembodiments, both the remote conference space 612 and the localconference space 610 contain LF display systems 600 as well as lightfield capture devices. In another embodiment, holographic data iscaptured in either space using one or more traditional 2D cameras alongwith a depth sensor. In still another embodiment, 2D image data iscaptured with one or more 2D cameras in either the remote or localconference space, and holographic data is generated from the 2D imagedata using 2D to 3D conversion techniques known in the art. In oneembodiment, the LF display modules 620 include bidirectional surfacesthat project a light field and simultaneously absorb incident light,relaying it to one or more image sensors, which may be used to record alight field from the area near the display surface. In the configurationwhere there is a light field display system 600 with such bidirectionalsurfaces located at both locations involved in the video conference,then there may be a one-to-one correspondence between the gaze directionfor participants in the physical locations connected by the videoconferencing solution because the LF display modules 620 are acting as aco-located display and camera. Thus, the participants may make eyecontact as if they were located within the same physical space. In otherwords, the LF display modules 620 act as a window between the twolocations, giving the impression that the participants are sittingaround a single table.

The LF display modules 620 may also include an audible sound fieldprojection system. This may emanate from electrostatic speakers that aremounted onto the surface of the display, with optically transparentmembranes, at locations between waveguide elements, co-located with orhelping form structures designed to inhibit light transmission betweenwaveguides, to allow both sound fields and light fields to besimultaneously projected from the display surface. The sound detected bymicrophones in one physical location may be emitted by corresponding LFdisplay modules 620 in the other physical location. Thus, when aparticipant in one physical location speaks or otherwise makes a sound,it may seem in another physical location as if the sound is coming fromthe hologram of the participant. Alternatively, separate microphones andspeakers may be used to establish an audio connection between thephysical locations.

In some embodiments, a haptic projection system is integrated into thedisplay surface formed by the LF display modules 620. Thus, the displaysurface may be a dual energy surface that projects both light fields andfocused ultrasound energy to create tactile surfaces. This may enableparticipants in a video conference to experience the sensation of makingphysical contact with remote participants, such as shaking hands. It mayalso be used to generate holographic objects that participants may touchand manipulate (e.g., holographic prop 650, which is described ingreater detail below).

In some embodiments, the display surface includes speakers (e.g.,electrostatic speakers) for generating audio fields. Thus, the systemmay control the direction from which participants hear sound. Forexample, the controller 520 of an LF display system 600 may use trackingsoftware to determine which remote participant is speaking and cause thespeaker to emit an audio field that substantially co-locates theapparent source of the generated audio and the holographicrepresentation of the participant who is speaking. This can furtherincrease the impression that the participants are all located in thesame physical space. Additionally or alternatively, the LF system 600may include one or more speakers that are outside of the display.

In the embodiment shown in FIG. 6, the conference space 610 alsoincludes a mobile LF system 660. The mobile LF system 660 may have adisplay surface area that is smaller than the display surface area ofone or more other LF displays, such as the surface formed with LFmodules 620. Although the mobile LF system 660 is shown as a robot onwheels, other forms of locomotion may be used. For example, the mobileLF system 660 may be mounted on tracks, mounted on an articulated robotarm, or be carried by a human participant. Regardless of the preciseform adopted, the mobile LF system 660 may provide functionality tofurther increase the impression than participants from differentphysical locations are present within the same space.

In various embodiments, the mobile LF system 660 may include an imagecapture system, an image display system, or both. The image capturesystem of a mobile LF system 660 may include one or more 2D cameras, adepth sensor, and/or a LF camera. Thus, a mobile LF system 660 may beconfigured to capture 2D image data and or holographic data for an areanear the mobile LF system 660 (e.g., in front of the mobile LF system).The mobile LF system 660 may also, in configurations that capture 2Dimage data, convert 2D image data into holographic data (e.g., usingdepth sensor data and/or 2D to 3D techniques known in the art). Themobile LF system 660 may also be configured to capture other types ofdata. For example, the mobile LF system 660 may include a microphoneconfigured to capture audio data.

The image display system of a mobile LF system 660 may include a 2Ddisplay, a LF display system, or both. In some embodiments, a mobile LFsystem 660 includes a bidirectional surface that may be configured tosimultaneously capture LF image data and project holographic content.

In one embodiment, the mobile LF system 660 is located in one space(e.g., the local conference space 610) but controlled by a participantlocated in a remote location (e.g., remote conference space 612). Thus,the controlling participant can navigate the mobile LF system 660 aroundthe conference space 610 to see perspectives that may not otherwise beavailable (e.g., the back of chair 640). Furthermore, the mobile LFdisplay 660 may create a holographic image of some (e.g., head andshoulders) or all of the controlling participant (or an avatar of thecontrolling participant). Consequently, the controlling participant mayexplore the conference space 610 and other participants who arephysically located within the conference space 610 may interact with theholographic image of the controlling participant as if they werephysically present. This may further increase the impression that theparticipants are located within a common physical space.

The LF display system 600 may also present supplemental holographiccontent in the conference space 610. The embodiment shown in FIG. 6includes two examples of supplemental holographic content: a holographicprop 650 (in this case, a holographic image of a car) and a holographicwhiteboard 670. The holographic prop 650 and the holographic whiteboard670 may be displayed in both the local and remote conference spaces, orjust one of these spaces. In other embodiments, different and/oradditional supplemental holographic content may be presented by the LFdisplay system.

The holographic prop 650 is a holographic object such as a 3D CAD model.The holographic prop 650 may be an image of a physical object (e.g., aproduct prototype located in the second conference space 612) or a CGvirtual object (e.g., generated from a file on a computer, such as thecontroller 520). This may further enable participants in differentphysical locations to interact as if they are in the same space. Forexample, a participant in the second conference space 612 may select a3D CAD model and the LF display system 600 generates the holographicprop 650 in a holographic viewing volume in the first conference space610. The location of the holographic viewing volume may be selectedbased on the preferences of one or more participants located in theconference space 610. Alternatively, the holographic prop 650 may appearat a default location.

In some embodiments, ultrasound is used to provide a tactile surface forthe holographic prop 650, as described previously. The tracking system580 of the LF display system 600 may track the motion of participants inthe conference space 610 to enable manipulation of the holographic prop.For example, a participant may be able to reach out and “grab” theholographic prop 650 to rotate and/or move it. The tracking system 580may also recognize certain gestures as commands relating to theholographic prop 650. For example, if a participant places both hands on(or near) the holographic prop 650 and moves their hands apart ortogether, the tracking system 580 may interpret these gestures ascommands to increase and decrease the size of the holographic prop 650,respectively.

If both conference space 610, 612 include a LF display system 600, ashared instance of the holographic prop 650 may be formed in eachconference space. Thus, interactions with the holographic prop 650 by aparticipant in one space can be reflected in the presentation of theholographic prop 650 in the other space. For example, if the holographicprop 650 is generated from a 3D CAD model, a participant in one spacemight rotate the model to make a particular feature visible whiledescribing that feature as part of a presentation. The holographic prop650 in the other space may be automatically rotated in the same wayenabling all participants to see the feature being described. In someembodiments, a participant may annotate the holographic prop 650 using astylus (or other such tool). The tracking system 580 may track themotion of the stylus and add the annotations to a file from which theholographic handout was generated and/or display holographic versions ofthe annotations in conjunction with the holographic prop 650. Returningto the previous example, the presenter might draw a circle around thefeature being described, provide written annotations providingadditional information about the features, add an arrow indicating howthe feature moves in operation, or the like.

The holographic whiteboard 670 provides an interface for participants todraw on, similar to a physical whiteboard. In some embodiments, the LFdisplay system 600 provides a holographic image indicating the extent ofthe holographic whiteboard 670 (e.g., a holographic image of a physicalwhiteboard) in each conference space 610, 612. For example, in theconfiguration shown in FIG. 6, each conference space 610, 612 mayinclude a holographic whiteboard 670 next to a physical table 630, 632.The holographic whiteboards 670 in each location are synchronized,meaning any additions or modifications made on one appear on the others.Thus, participants in different physical locations may collaborate onthe holographic whiteboard 670 as if they were present in the samephysical space. Ultrasound may be used to create a tactile surface forsome or all of the holographic whiteboard 670 to gives participants thesensation of drawing on a physical whiteboard (or some other surface,such as paper, a chalkboard, canvas, etc.).

In one embodiment, the contributions of different participants arestored as different layers. A participant may select which layers aredisplayed on the holographic whiteboard 670 (e.g., using a physicalremote control and/or holographic controls). The layer selection may besynchronized across instances of the holographic whiteboard 670.Alternatively, participants in each location may independently selectwhich layers to display. In other embodiments, layers may be added,used, and displayed in different ways. For example, participants may beprovided with controls (e.g., physical or holographic) to add and deletelayers as well as select which layer is currently active. New contentmay be added to the currently active layer. The controls may also enableparticipants to split layers and move content between layers. Thus,participants may control what content on the holographic whiteboard 670is in what layer.

In some embodiments, the holographic whiteboard 670 is 3D. Because thewhiteboard is holographic, it need not be limited to a 2D surface.Participants may draw lines that move forwards/backwards as well asup/down and/or left/right. This may enable participants to directly draw3D structures and more easily express complex concepts and relationshipsthan is possible when limited to 2D representations. In one embodiment,a 3D whiteboard is divided into slices along the forwards/backwardsaxis. Participants may select which slices are displayed to viewdifferent cross sections of depicted 3D structures.

In some embodiments, the holographic whiteboard 670 may be used todisplay 3D holographic objects such as CAD models, building schematics,park plans, etc. The 3D objects may be generated from a data file or byimaging a physical object (e.g., a product prototype, set of blueprints,etc.) located within one of the conference spaces (e.g., remoteconference space 612). In one example, each page of a document may bepresented as a different layer on the holographic whiteboard 670.Participants may select which layers to view to read specific pages.Participants may also move pages around within the 3D volume of theholographic whiteboard independently. Thus, a participant may view anycombination of pages (e.g., side by side). Similarly, a participantcould arrange corresponding pages of two different versions of adocument to enable easy comparison. As another example, the LFprocessing engine 530 might take a set of blueprints for a building anduse them to create an approximate 3D model of the building. Participantsmay then interact with the holographic whiteboard 670 to, for example,view different cross sections of the building, remove certain elements(e.g., wiring, doors, specific walls, etc.) from the displayed model,make modifications to the model (which may then be used to generateupdated blueprints), add annotations, etc.

The LF display system 600 may modify the images of participants and/orother holographic objects. In one embodiment, to improve the impressionthat holographic objects are located within the conference space 610,the LF display system 600 determines lighting parameters (e.g., theoverall brightness and spectral distribution) within the conferencespace 610 and adjusts the holographic images to better match thelighting parameters. For example, if the conference space 610 has abright light on the left-hand side, the holographic images might beadjusted such that the left side is brighter while the right side is inshadow. This may increase the impression that the images are physicalobjects that are located within the conference space 610.

The LF display system 600 may also make adjustments to audio parametersfor generated audible sound to account for acoustic effects within theconference space 610. For example, if one corner typically has muffledsound, the LF display system may apply an equalization boost to a highfrequency band of the acoustic energy directed towards that corner toprovide clearer sound. The LF display system may also provide noisecancelling. For example, one or more LF display modules 620 at the backof the conference space 610 may emit sound waves that partially orcompletely cancel soundwaves emitted from the LF display modules 620 atthe front of the conference space 610.

Light Field Display Video Chat System

FIG. 7 shows an alternative configuration for a conference space 710including a LF display system 700, in accordance with one or moreembodiments. The LF display system 700 is an embodiment of the LFdisplay system 500. In FIG. 7, rather than generating holographic imagesof a table and chair located behind the display surface, an LF displaysystem 700 generates holographic images of a table 730 and chair 742 infront of the display surface. Thus, the physical table 730 and chair 740in the conference space 710 are not positioned against the LF displaymodule 720 as they were in FIG. 6. To do this, the LF display systemtranslates the holographic images by projecting them such that theyappear in front of the LF display modules 720, rather than behind them(as they did in FIG. 6). Similarly, in the other physical location(where the table 732 and chair 742 are located), holographic images ofthe table 730 and chair 740 may be formed by translating those images byprojecting them such that they too appear in front of the displaysurface of the other physical location.

Translating the images in this way may improve the impression that allof the holographic images (including participants) are located in thesame physical space. Rather than the LF display modules 720 acting as awindow through which participants can see but not pass, it is projectingimages of the remote participants into the conference space 710. Thus,participants way walk up to and around each other. Furthermore,ultrasound tactile surfaces may be used to simulate participants inremote locations touching each other. Although not shown in FIG. 7, theconference space 710 may also include additional holographic objects,such as holographic handouts, whiteboards, and the like, as describedabove with reference to FIG. 6.

FIG. 8A is an illustration of a LF display system 800 presentingholographic content including a holographic video chat participant, inaccordance with one or more embodiments. The LF display system 800 is anembodiment of the LF display system 500. The LF display system 800 hasLF display modules 820 of a LF display assembly forming a one-sidedseamless surface environment. The LF display system 800 provides videochat functionality in which a participant views holographic images ofsome or all of the other participants, generated by the LF displaymodules 820. The LF display system 800 may also include any combinationof the other components of the LF display system 500 such as the sensoryfeedback assembly 570, the tracking system 580, the viewer profilingmodule 528, and the controller 520. In other embodiments, the LF displaysystem 900 includes additional cameras separate from the tracking systemfor capturing image data.

In the illustration of FIG. 8A, a first video chat participant 830 inlocated within a viewing zone of the LF display system 800. The LFdisplay system 800 captures image data of the first participant 830 viaany combination of a tracking system (e.g., a tracking system 580), theLF display modules 820, one or more cameras included in the LF displaysystem 800, and any additional tracking devices. In some instances, theLF display system 800 receives image data of the first participant 830from multiple perspectives such as from one or more LF display modules820 or from one or more cameras separate from or as part of the trackingsystem. In some embodiments, the LF display system 800 has camerasaround the first participant 830 which capture image data encompassingperspectives over all 360°. Alternatively, the LF display system 800 maybe “one way,” meaning the first participant 830 is presented with aholographic representation of the second participant 840, but the LFdisplay system 800 does not collect image data of the first participant830 and/or does not provide such image data to the display device beingused by the second participant 840.

The LF display system 800 produces holographic content that includes aholographic representation of a second video chat participant 840. Theholographic representation of the second video chat participant 840 is aholographic image. In one embodiment, the LF display modules 820 are abidirectional surface that also collect image data for the firstparticipant 830 and send it to a LF display system of the secondparticipant 840 for presentation as a holographic image. In other words,the LF display modules 820 may act as a window through which theparticipants may view each other. As with the video conferencing systemdescribed with reference to FIG. 6, there may be a one-to-onecorrespondence between gaze directions between participants, enablingthose participants to make eye contact with each other as if they werelocated in the same physical space. Alternatively, the LF display system800 may be “one way” with only one of the participants being presentedwith a holographic representation of the other.

FIG. 8B is an illustration of the LF display system 800 of FIG. 8Apresenting holographic content including a holographic image of thesecond video chat participant 850, in accordance with one or moreembodiments. Similar to the video conferencing configuration illustratedin FIG. 7, the LF display system 800 is projecting the holographic imageof the second participant 850 into the same physical space as the firstparticipant 730. Thus, in contrast to the example shown in FIG. 8A, thefirst participant 830 experiences the impression that the secondparticipant 850 is in the same physical location as them. The same orsimilar techniques as described previously for adjusting lighting and/orsound parameters to improve the overall experience may be applied.Furthermore, as described previously, an ultrasound projection systemmay be used to create volumetric haptic tactile surfaces for some or allof the holographic representations of the participants. These tactilesurfaces may simulate the participants touching each other, furthergiving the impression that they are located within the same physicalspace. The ultrasound projection system may work in conjunction with thetracking system 580 to update the tactile surfaces. For example, usingthis approach, remote participants in a video chat may experience thesensation of shaking hands with each other, with the haptic tactilesurfaces each participant experiences being updated based on the motionof the other participant's hand.

The LF display system 800 may also enable video chat participants tomake changes to their appearance and/or voices. In one embodiment, theLF display system 800 provides controls (e.g., physical or holographic)enabling a participant to adjust parameters of the holographicrepresentation of them presented to other participants. For example,participants may blur out their own faces, select a custom background,add balloons or other objects floating around them, make it appear likethey are sitting behind a desk, or the like. Similarly, the LF displaysystem 800 may enable participants to apply filters to alter theirvoices, such as increasing or decreasing pitch, making them sound like arobot, selectively reducing the intensity of certain frequencies, or thelike.

In other embodiments, the LF display system 800 provides controls (e.g.,physical or holographic) enabling the participants to apply filters tochange aspects of their own or other participants' appearances. Forexample, a filter may change a participant's clothing, change aparticipant's hair color, replace a participant's image with an avatarmapped to that participant's movements, and/or apply any other visualfilter to how a participant appears. In one embodiment, the filters aremodular and participants may install (e.g., by downloading from awebstore, etc.) those that they wish to use. In some instances,additional filters may be made available by third parties. Suchthird-party filters may be free and/or made available for a price (e.g.,through a marketplace).

FIG. 9 is an illustration of a LF display system 900 presentingholographic content including holographic representations ofparticipants in a group video chat, in accordance with one or moreembodiments. The LF display system 900 is an embodiment of the LFdisplay system 500. A first video chat participant 930 is in the samephysical location as the LF display system 900. The LF display systemincludes LF display modules 920 that are generating a holographic imagerepresentation of an active participant 950 in the voice chat. The LFdisplay modules 920 are also generating holographic imagerepresentations of one or more additional participants 960. Therepresentations may be images of the participants and/or avatars of theparticipants. In FIG. 9, two additional participants 960 are shown, buta voice chat can include any number of additional participants(including zero). Furthermore, although the representations of theadditional participants 960 are shown as being reduced in size, in otherembodiments, other indicators may be used for the active participant950, such as displaying the representation of the active participant 950in a particular location (e.g., centrally), displaying a particularvisual indication in conjunction with the representation of the activeparticipant 950 (e.g., a glowing above the representation of theparticipant's head, etc.), or any other suitable indicator. In someembodiments, not all participants are represented by holographicrepresentations. For example, participants who do not have access to aLF camera (or who choose to not use it) may instead be represented by astatic avatar.

The LF display system 900 may determine which participant is the activeparticipant 950 using any suitable method. In one embodiment, the LFdisplay system identifies a participant that is currently speaking asthe active participant 950 based on the collected acoustic energy. Inanother embodiment, the first participant 930 may change the activeparticipant 950 using controls provided by the LF display system 900(e.g., by pointing at one of the holographic representations of theadditional participants 960). Although FIG. 9 illustrates theholographic representations as standing, this need not be the case. Forexample, the representations of the participants might be presented assitting around a holographic (or physical) table, with the activeparticipant 950 being placed at the head of the table.

In various embodiments, an LF display system 500 provides a holographicvoice mail function. This operates substantially as described above withreference to video chat, except that the LF display system 500 creates amessage for later presentation by collecting audio and visual datacorresponding to an individual and saving it (e.g., in the data store530) rather than presenting it substantially in real time at anotherlocation. For example, the individual may use physical and/orholographic controls to begin, pause, and end recording and then selectone or more individuals to receive the recorded message. At a latertime, a recipient may use physical and/or holographic controls of a LFdisplay system 500 (which may be a different or the same system thanused to record the message) to trigger playback of the message. Duringplayback, the LF display system 500 presents a holographic image of theindividual who recorded the message along with the corresponding audiocontent. In some embodiments, the same or similar techniques describedabove for modifying an individual's appearance and/or voice may be used.

In one embodiment, the LF display system 500 provides controls to enablea recipient to control playback of the message, such as controls tobegin, pause, fast-forward, rewind, and/or end playback. As the LFdisplay system 500 creates holographic content from the visual data, therecipient may move around to view the visual content of the message fromdifferent perspectives. Alternatively, the controls may include controlsfor rotating, translating, zooming in and out, and the like so that therecipient may view different perspectives without moving. Thus, therecipient may play the message multiple times to view it from differentperspectives.

In some embodiments, similar playback functionality may be provided forrecordings of video conferences and/or video chat sessions. The LFdisplay system may provide an option for a video communication sessionto be recorded. Participants (and anyone else given access to therecording) may then replay the recorded session to view the recordedscene from different perspectives.

ADDITIONAL CONFIGURATION INFORMATION

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations of LF display system are possible in lightof the above disclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In some embodiments, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the disclosure be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

1.-39. (canceled)
 40. A light field display system in a local physicalspace, the light field display system comprising: a controllerconfigured to generate display instructions based on visual datacorresponding to a remote scene, the visual data received from a remoteimage capture system located in a remote physical space; and a locallight field display assembly located within a local physical space thatincludes a local participant, the light field assembly including abidirectional surface configured to simultaneously capture image datacorresponding to local physical space and project holographic contentgenerated from the display instructions, the holographic contentincluding a holographic video of the remote scene, wherein thecontroller is further configured to send, to a remote image displaysystem in the remote physical space, outgoing visual data derived fromthe captured image data; wherein the remote image display system isoperable to generate video content including the local participant basedon the outgoing visual data.
 41. The light field display system of claim40, wherein the local light field display assembly comprises a pluralityof light field display modules.
 42. The light field display system ofclaim 41 wherein the plurality of light field display modules are tiledto form a seamless display surface.
 43. The light field display systemof claim 40, wherein the remote image capture system is a remote lightfield display system that includes a bidirectional surface thatsimultaneously captures light field image data corresponding to theremote scene and projects the video content including the localparticipant.
 44. The light field display system of claim 40, wherein theholographic content further includes a holographic prop.
 45. The lightfield display system of claim 44, wherein the holographic prop is aholographic image generated from a 3D model of an object.
 46. The lightfield display system of claim 44, further comprising a tracking systemconfigured to generate tracking data representing motion of the localparticipant, wherein the holographic prop is manipulated based on thetracking data.
 47. The light field display system of claim 40, whereinthe holographic content further includes a holographic whiteboard,wherein content on the holographic whiteboard is synchronized with acorresponding holographic whiteboard generated by a remote light fielddisplay assembly.
 48. The light field display system of claim 40,wherein the display instructions are operable to further cause the locallight field display assembly to emit energy to create a tactile surface.49. The light field display system of claim 48, wherein the tactilesurface is collocated with at least a portion of a holographic image.50. The light field display system of claim 49, wherein the holographicimage is a holographic image of a remote participant or a holographicprop.
 51. The light field display system of claim 40, further comprisinga mobile light field system located in the local physical space.
 52. Thelight field display system of claim 51, wherein a remote participantlocated in the remote physical space controls motion of the mobile lightfield system within the local physical space.
 53. The light fielddisplay system of claim 52, wherein the mobile light field system isconfigured to capture image data corresponding to an area near themobile light field system and provide the captured image data to theremote image display system.
 54. The light field display system of claim51, wherein the mobile light field system includes a bidirectionalsurface configured to simultaneously capture light field image data andproject holographic content.
 55. The light field display system of claim54, wherein image data captured by the mobile light field system isprojected as holographic content to a remote participant.
 56. The lightfield display system of claim 55, wherein a hologram of at least aportion of the remote participant is projected in the local physicalspace simultaneously with the holographic content being projected to theremote participant.
 57. The light field display system of claim 40,wherein the holographic content includes a holographic image of a remoteparticipant in the remote physical location and one or more holographicimages of one or more additional remote participants located in one ormore additional remote physical locations.
 58. The light field displaysystem of claim 40, wherein the visual data is vectorized and the lightfield display system further comprises a processing engine configured toconvert the vectorized visual data into rasterized visual data, thedisplay instructions being generated from the rasterized visual data.59. The light field display system of claim 40, wherein the light fielddisplay system further comprises a local decoding block that matches aremote encoding block used to compress the visual data, the localdecoding block configured to decompress the visual data.
 60. The lightfield display system of claim 59, wherein the captured image dataincludes visual data corresponding to the local participant and thelight field display system further comprises: a local encoding blockconfigured to compress the visual data corresponding to the localparticipant; and a network interface configured to send the compressedvisual data corresponding to the local participant to the remote imagedisplay system.
 61. The light field display system of claim 40, whereinthe controller is further configured to receive additional visual datathat corresponds to a prerecorded holographic video and generateadditional display instruction, the additional display instructionscausing the local light field display assembly to generate theprerecorded holographic video responsive to user input provided by thelocal participant.
 62. The light field display system of claim 61,wherein the light field display system further comprises controlsconfigured to enable the local participant to change a perspective fromwhich the prerecorded holographic video is viewed.